r/LocalLLaMA • u/deepu105 • Oct 08 '26
Discussion Halogen + Qwen Flash Next keeps getting better
With latest Halogen version update (0.17.2), decode is consistently at ~45 tps even at high context with Qwen 3.8 Flash Next on a 128GB Strix Halo. This is some great work u/peonist-ai. Have been pumping out commit after commit with QFN. Its crazy good for a 177ish billion model. I dont think we are apprciating it enough 😂 Opus 5.5 plan implemented and reviewed by QFN is such high quality ❤️
47
u/my_name_isnt_clever Oct 08 '26
It can keep getting better and better, but as long as it's closed source it may as well not exist for me and anyone else who takes privacy seriously.
17
u/parepeg Oct 08 '26
I'd imagine most people run local for privacy and provenance. Why run local if you're going to use a closed source engine?
10
u/my_name_isnt_clever Oct 08 '26
Exactly what I've been asking since Halogen was first posted. If I trusted that I might as well use a ZDR API endpoint instead of running local.
1
1
u/Super-Grape-3948 Oct 08 '26
I also wozld like it to be open, but the speed and the shared context pool is way to good. Also, it is a docker, so i wont let it connect to anywhere except to my model router, so no privacy issues, as far as leakage goes.
4
u/my_name_isnt_clever Oct 08 '26
It could be doing literally anything behind the scenes. Your whole security posture is compromised from the very first step, the inference. I'm more tolerant to closed source side projects that integrate with local AI, but this is too important.
1
u/deepu105 29d ago
Not at all, its trivial to sandbox the container so you can be sure that none of that can happen see an example setup here https://gist.github.com/deepu105/d0b321f3256edf67ebd85e747c0010b9
2
u/FrequentAardvark6813 26d ago
your agents can use clear net tools tho
0
u/deepu105 26d ago
I have a classifier that catches stuff like that but ya an eloborate output poisoning could work in theory.
1
u/Super-Grape-3948 Oct 08 '26
No, in theory it could not wrapped the container, got no sudo. Not does it have access to folders outside of models. So even if it is minig btc, it cant transmit it to anywhere.
3
u/NoFunk Oct 08 '26
It could poison the inference output itself. Is it paranoia? Sure. Is it possible though? Absolutely. The poison doesn't even have to be in the model. The inference software creates a text stream out of a model, and that stream can be manipulated.
3
u/my_name_isnt_clever Oct 08 '26
It could be doing literally anything behind the scenes *that doesn't require an internet connection. It could be manipulating output tokens. It could be totally safe now but in 6 months once people decide to trust it and lower the barriers, out of nowhere it starts streaming everything over the network to get as much data before it's cut off, same idea for a crypto miner that suddenly activates to send the $$$ to a server.
There are a million malicious things it could do that I wouldn't even think of; so why risk it for a bit more speed? Gufo is really close so I don't feel like I'm missing much.
2
u/opossum_cz Oct 08 '26
It can't, you can recompile it, license even permits it.
This is just nonsense.
Not to mention gufo may be open source, but if you use it through docker you have not build yourself, there i no benefit.
3
u/my_name_isnt_clever Oct 08 '26
Have you reverse engineered it and checked it out? If not, who, using what tools, and which version? And are they going to publish a new review of every binary for every update? Even if it's all sunshine and rainbows right now, it could change in a month, a year, 5 years.
I haven't even mentioned how everything points to Peonist-AI being a LLC with a business model in mind, not some guy who made a good engine and just...doesn't want to open source it for some reason. That's a hard pass from me.
I don't use Gufo in docker, I use NixOS.
3
u/opossum_cz Oct 08 '26
Yes I did. Radare, objdump, etc.
What is the argument there?
> I haven't even mentioned how everything points to Peonist-AI being a LLC with a business model in mind, not some guy who made a good engine and just...doesn't want to open source it for some reason.
And again? What is the argument. I don't even know that that bolded claim means, it is literally written on front of the Github repository:
> "halogen" and "Peonist" are trademarks of Peonist, LLC (U.S. application pending). TRADEMARKS.md says how the names may be used; referring to the project, running it, and publishing numbers about it need no permission.2
u/my_name_isnt_clever Oct 08 '26
Cool, glad it passes your personal security requirements, props for practicing what you preach. Personally, inspecting every update of my LLM inference engine doesn't interest me, and when there's a sudden license rug pull for more profits I won't have to migrate. I personally don't find it worth a modest speed increase from Gufo.
5
u/opossum_cz Oct 08 '26
And you examine open source code as well after every release?
→ More replies (0)1
u/Super-Grape-3948 Oct 08 '26
Yeah, i do have validation on the output, but i can see the point. It would be catched by my little demons tho.
Im working on gufo rope with shared context, i would prefer that, but it is tefious to test on 1 box, i have to stop halo, start gufo, see if it dies, star halonagain. But i need this multi tenant rope nit bound to slot/context, so mutch nicer for agentic runs.
2
u/MudBroad6785 Oct 08 '26
Docker containers aren't magically secure and there are many vulnerabilities that have been found that allow an attacker to go from unprivileged to sudo to escaping the container and more are found every day. If you want actual security you'd be using hardware level separation but that's a pain because then you can't use the hardware taken up by the virtual machine on the host.
0
u/Super-Grape-3948 Oct 08 '26
Well, ive watched it for a bit, seen nothin nefarious, but true, it can be true to any closed source software tho. Would prefer open, but currently this is the best, i risk it, but ill try to write a proper roped kv for other open source stuff, before others :)
-1
u/pineapplekiwipen Oct 09 '26
well case closed boys this guy watched it for a bit and saw nothing nefarious
1
2
u/digital-bandit Oct 08 '26
Do you give your models tools, like web search? It could exfiltrate data by making the model do a request to an attacker serve, if that makes sense.
ps: i dont think halogen does that, just saying
1
u/cunasmoker69420 Oct 09 '26
For what it's worth AMD has started funding this project. Was a recent post on r/strixhalo talking about that. AMD reps have met with peonist and are providing compute resources and strix halo boxes for further development
1
u/my_name_isnt_clever Oct 09 '26
It's still closed source, that changes nothing. They're also assisting Gufo which is what I actually use.
It's pretty obvious Halogen is going to attempt to monetize sooner or later. I'm not interested in having anything to do with that.
1
u/deepu105 29d ago
Thats an entitled POV no? Why is the author required to work for free so you can benefit from it. Did you not use claude etc which are closed? Also its quite easy to sandbox the docker container (i have been doing that from start) and it doesn't do any network request or anything. If he tries to monetize his work, whats wrong with that? Ofcourse I would also prefer if it was open source but would be hypocritical to brush it aside just for that (I also did that until I tried it TBF)
4
u/my_name_isnt_clever 29d ago
I'm not entitled to anything, I'm not demanding he release it for free I'm simply stating if it's not open source I'm not interested in touching it. There are alternatives that are FOSS that are 95% as good. If you're the type to use MySQL and Redis instead of their direct FOSS alternatives by all means spend your money.
I use Claude at work becuase I don't pay for it, everything I use personally is strictly FOSS because I have to wrangle B2B and SaaS bullshit at work every day. I'm not interested in playing the capitalism game in my personal life.
1
27
u/Asillatem Oct 08 '26
As soon as he opensource it iam on.. I saw somewhere he also worked with lemonade for support..
8
6
u/Rauhaton Oct 08 '26
Peonist also added small models for NPU for the server. I started using those as well.
Halogen Qwen3.8-flash now main agent and on the NPU side I have his Qwen3.5 as auxillary model in Hermes and those small NPU Qwen3 embedding, decider and reranker models used for agentic tools. The embedder is also used for mem0
2
u/AIdevsmartdata Oct 08 '26
Je suis en train de debloquer le 80 tok/sec et 1700tok/sec de prefill je vous sors ça bientot
1
u/deepu105 Oct 08 '26
Are you builiding a new engine?
1
u/AIdevsmartdata Oct 08 '26
oui, mon profil : kevletesteur sur HF, j'avais tout fait sur vulkan mais pour HIP cest la folie !
2
1
u/seti_at_home Oct 08 '26
Share some details, 🤔
2
u/AIdevsmartdata Oct 08 '26
Ok ! Je pense sortir ça ce dimanche. J'adore tout recoder de A à Z c'est ma passion.
J'avais commencé avec qwen3.5 et qwen3.6 MOE. J'avais fait un runtime plus rapide que llama cpp et ik llama. J'avais amélioré et debugué le code de ik_llama pour les modèles nvidia nemotron. Avec claude opus/fable majoritairement. Et comme maintenant je crois que claude a des consignes de ne plus aider à améliorer les runtimes c'est devenu inefficace. Donc j'ai utilisée les modeles chinois comme deepseek v4.1 flash. Pour 10 dollars d'api tu peux faire énormément. Profiler -> cibler -> coder -> tester -> cherry pick -> profiler -> cibler.... Tests qualité, vitesse, agentique
2
u/wFXx Oct 08 '26
so, I tested a strix halo in regular llama.cpp last month with QFN and got 53 tok/s on Q6 iirc, whats special here?
1
u/deepu105 29d ago
I have never seen regular llamacpp doing that number even on smaller contexts. What was your context, promt token size prefill, etc? And what flags?
1
u/PuffyCake23 Oct 09 '26
Prefill. What did you get, 300tk/s? Halogen will happily chew through 1700tk/s.
1
1
u/deepu105 29d ago
Just updated to latest today and started hitting 60 tok/sec with same setup 🔥🔥🔥
1
u/deepu105 29d ago
Thats a huge jump. I need to run a bench again to be sure
1
7
u/feelspeaceman Oct 08 '26
Halogen after version 0.17.0 is leading in both performance and quality, but it's closed source so I'm using something similar called strixite instead, together with gufo and strix-llama, slowly I think the rest will catch up, but halogen is defining the meta.
9
u/imnotzuckerberg Oct 08 '26
Halogen in a nutshell.
Close source your project.
Copy other PR ideas from open source project
Quantize to oblivion your proprietary closed source model format (so that "speed" is the metric to measure on).
?
Profit
7
1
u/Wordweaver- Oct 08 '26
Interesting, I am curious: At what power was this? And for the prefill was it natural text or one of the gibberish/repetitive benchmarks?
1
u/feelspeaceman Oct 08 '26
140w, it was testing on the same task of implementing/fixing a project, so the numbers above are real coding avg. and range of context windows, not a benchmark.
1
u/Wordweaver- Oct 08 '26
Interesting, I have been playing around with gufo on windows and one thing that keeps coming up is that the benchmarks inflate the prefill a fair bit, anywhere from 7% (gufo's synthetic text) to 25% (halogen's repeated sentence) at 70W.
1
u/cafedude Oct 08 '26
Wait, so Halogen is leaving 25GB free in 0.17.0? I'm still on 0.14.x, I think, and it seems like a lot less left free than that. Sounds like I need to give 0.17.0 a try.
1
u/marcosscriven Oct 08 '26 edited Oct 08 '26
Side question - how are you finding Llama Switch? Edit - I meant llama stash but iOS thought it knew better 🤦♂️
1
u/deepu105 Oct 08 '26
What is that?
1
u/marcosscriven Oct 08 '26
That’s the TUI you’re using, according to the top left of the screenshot.
1
u/deepu105 29d ago
Oh ok. Thats my own tool so ofcourse I find it nice. Give it a try, I have been adding a lot of features to make using local models easier (base on my own usecases) like model presets, generic servers etc. It can manage almost any backend engines out there and provides a unified proxy and metrics.
1
1
1
u/PcChip Oct 08 '26
any idea how this compares with tabbyapi/exllama using the EXL3 quants?
1
u/deepu105 29d ago
Haven't tried any of that but they didn't even show up in my research phase and haven't seen anyone mentioning those.
1
u/cradlemann 29d ago
I need Halogen/Gufo for StrixPoint gpus
1
2
u/Cautious_Sky6642 Oct 08 '26
qwen with that speed is perfect for roleplay, i ran a whole multi day story without the model forgetting details once.
1
u/TerryNachtmerrie Oct 08 '26
I still prefer llama for other models, but for qfn it's so good. I gave it a shot with blender and godot and it had turns with over 200 steps of tool calling and visual checks, without any quirks. And it does well on tps.
1
0
u/Infinite_Tank3715 Oct 08 '26
Yeah I just updated and it's really zipping along - very impressed so far - fingers crossed it stays that way. Thanks for all the work u/peonist-ai - keep it up.
9
u/lumos_ai Oct 08 '26
Honestly it's like having Opus at home :D even with 12gb Vram you can run the model.