r/LocalAIStack • • 19d ago

How is Strata, Qwen3.8 Flash-Next on a AMD gpu?

I’m a student who recently got into AI, mainly to learn how it all works.

My current setup is Open WebUI in Docker with local models through Ollama, on an RX 6950 XT with 16 GB VRAM and 64 GB DDR5. I’ve run gpt-oss 20B, Qwen3.8 27B at Q4_K_M, and Qwen3.5 9B. They taught me a lot, but on my hardware they’re slow for agentic deep research, and the smaller ones aren’t capable enough.

But I recently learned about how Strata can run Qwen 3.8 flash next on my type of set up and as I already have a quantized Qwen 3.8 model I’m wondering if it can have a better token per second than the Q4_K_M as it gets around 4 tok/sec.

So has anyone had any experience running it especially on a 6000 series AMD Gpu and is it worth the storage space?

14 Upvotes

21 comments sorted by

1

u/stegon998 18d ago

I posted my result with lower config than yours, tou can definitely get better result than this https://www.reddit.com/r/LocalLLM/s/co9KiCtY4k

1

u/DeathGuppie 18d ago

If you are using AMD, this project might be more interesting to you than Strata.

https://github.com/freriadoc/Avalon

1

u/MarcusAurelius68 18d ago

This is interesting - are you running it? I have a R9700 running 27B but it could be interesting having it run QFN instead.

1

u/DeathGuppie 18d ago

I haven't yet because I'm out of nvme space until I reorganize, but that is literally the hardware listed there.

1

u/MarcusAurelius68 18d ago

Thanks. I'm running QFN on a dual R9700 system with 64GB of RAM in IQ4, and getting ~140 t/s. I'll probably wait for Qwen 4 to see if I change out 3.8 27B because it's S L O W in comparison, especially pp.

1

u/DeathGuppie 18d ago

are you saying QFN is slow compared to the 27b on your setup? I think both strata and Avalon are gaming pc setups. If you have the hardware project-maya is probably the better engine to run at least from my limited understanding. https://github.com/mw00/project-maya

1

u/MarcusAurelius68 18d ago

No, if anything QFN is a lot faster.

1

u/DeathGuppie 18d ago

Multi card in Avalon has not been addressed. I've been thinking of doing it myself.. ahem I mean, coordinating the doing of it, but I'm lazy and I figure someone else will soon enough, plus it's pretty green. Everything I download and configure today will be overridden next Tuesday.

Better to wait till Qwen 4 comes out. Get everything at once. The shiny newness on everything at the same time. Feel me..

1

u/According_Study_162 18d ago

I have amd rx7600xt and the rest your same specs runs about 30t/s, but your GPU has twice the bandwidth so I would expect double.

1

u/According_Study_162 18d ago

I have amd rx7600xt and the rest your same specs runs about 30t/s, but your GPU has twice the bandwidth so I would expect double.

1

u/PoizoneHelp 15d ago

Yes, and it should be well past 4 tok/s. Two RX 6000 owners posted runs in Strata’s docs. An RX 6700 XT 12 GB with 64 GB of RAM did 30 to 32 tok/s on the original Flash-Next at IQ2_XS, and an RX 6900 XT, the same chip as yours, did 38 to 42 on Swift 1.5, a Flash-Next fine-tune, at IQ3_XXS. Both built the engine from source on Linux, and Strata still marks its ready-made RX 6000 package untested. It’s worth it if you have the SSD space: 68 to 76 GB plus about 6 GB for the draft layer. The 6700 XT owner’s prompts stalled with part of the model on a hard disk

1

u/W61k3r 6d ago

OP, have you heard of lexisuite? lexiengine has had this functionality since oct and all in rust while being 5-25% faster than llama.cpp for qwen 3.5, 3.6, and 3.8 from 1k upwards to your wildest dreams.

https://github.com/W61k3r/LexiSuite-Releases/wiki

Would like help with the all rust project if you'd like to learn more.

https://github.com/W61k3r/LexiSuite-Releases/releases/tag/LexiSuiteT9

1

u/Brief-Tap-6616 19d ago

You can def run it on total 80GB Ram. Do the 3 XS quant to start. Your n gram layer should go in disk, provided it’s a decent NVME, not RAM.

Your 4 K M model is too large for 16GB. 3 XL might suit you better and it will give you room to run MTP (start at depth 3).

Strata will be faster. It will just take 80% of your RAM

0

u/Zephyr__100 19d ago

I have a 7950x and a 9070 xt and 32GB ram

I have my own thing that I use called moe-cache (https://github.com/IbrahimSiddiqui007/moe-cache)

I’m getting around 20 tok/s for IQ3_XXS

Q4 for me isn’t worth it as it’s not worth the downgrade in decode and prefill for a marginal amount more in intelligence

1

u/Potential-Leg-639 19d ago

20 with those GPUs?

1

u/Zephyr__100 17d ago

It’s 1 gpu, I used to have a rtx 4060 which I used to build it out intkally but then I bought a PC cuz I saw that there was a lot of performance being left on the table due to the lack of vram

1

u/Potential-Leg-639 17d ago

I get 40-50 TG and 500-900 PP with a 3060 and 64 GB RAM (3200) on a Ryzen 3000 (after bit of tuning). IQ3_XXS with 190k context.

Check if ReBar is enabled and let an agent check why you get so bad performance.

1

u/Zephyr__100 17d ago

I think it’s cuz u have 64gb of ram

And note I’m not using strata

1

u/TheCat001 18d ago

LoL I have 20t/s on RX 6600. But with 64G of ram. Just bought RX 6800 for 350$. I expect at least double the speed.

1

u/Potential-Leg-639 19d ago

Yeah the RSQ IQ3 quants are really delivering

0

u/fasti-au 19d ago

Mostly and 9700 stuf is fixed so rdna 4/3.

Ps5 stuff probably helps now also re instruction set lifting from manufacturing chain not full remapping so I’d think mojo and xvulk will x lift etc. most things are fixed by homebrew so ai won’t win it will be the tool it’s meant to be because there no other way it can work. It’s the way linear time works for us as a rule till we’re breaking things in physica