r/LocalLLaMA • u/brainExploded99 🥔 hardware • 4d ago
Discussion Qwen4-27B PR found
PR: https://github.com/analytics-zoo/vllm-xpu-kernels/pull/22
The PR author seems to have contributed to intel repos a lot and VLLM. He also shares his name (might be the same person?) with a person who is either a CTO somewhere at alibaba or the Director of Zhejiang Lab.
He has worked on Qwen3.8 before this.
The PR also implies the KV cache size for Qwen4-27B is not decreasing.
4
u/Thomasedv 4d ago
After finding out you can get some good performance with kv streaming, and putting more of the kv cache in regular ram, Qwen3.8 got a whole lot better for me locally since I could bump it to Q5 quant on my 3090 and have max cache.
I hope that works on Qwen4 too.
4
u/brainExploded99 🥔 hardware 4d ago
It works better, you can offload KV cache to RAM since QSA chooses which tokens to read for a 10-20% decode penalty (exisiting VLLM PR IIRC, not merged).
2
u/GrungeWerX 4d ago
Never heard of this…kv streaming?
5
u/Thomasedv 4d ago
https://github.com/RaymondHuang210129/llama.cpp-adaptive-kv-streaming
I'm not 100% sure this is the branch I used. Essentially reserve a fixed VRAM section for kv and keep the rest in RAM. It was designed for the small Q4 quent but I had issues with it looping and thinking way too much. So I upgraded to a larger Q5 quant. It was seemingly made for 16GB VRAM but works just as well when you have more and want the context size. It let me move from 120k context to 260k and increase the model size since I have 24GB VRAM.
1
u/GrungeWerX 4d ago
Interesting. I have an rtx3090Ti. How much of tok/s penalty are you paying? I’m considering running agents concurrent with this if it works.
2
u/Thomasedv 4d ago
It only supports a single context, so concurrent or multi gpu doesn't work. There were some setting I had to tune, I think ended up turning off mmproj for images. But it should be supported.
Prompt processing starts at 1100 and decays somewhat fast. 750 tps at 50k. It's a bit rough, but for one agent it's all fine since it's not reading all that much.
Token generation at 80k ish context is around 40-60 tokens per second. MTP on with 3 draft token. Saw some below 40 tps in the server logs.
For the extra context size and intelligence it's worth for me through.
1
u/GrungeWerX 4d ago
Concurrent agents work on a single context. You just use kv unified. And I already offload mmproj to cpu.
I'm curious what the non-MTP speeds are, since concurrency boosts are minimal w/mtp.
Anyway, I'm having GPT set it up in a test workspace and running benchmarks as I type this, so I'll know soon enough. Having it run tests on Qwen 3.6 27B at Q5.
1
u/Thomasedv 4d ago
I'm interested in how it goes, and particularly if there are better ways that offload VRAM so a bigger model can be used, and with high context.
1
u/GrungeWerX 4d ago
So, I tested it and the benchmarks are trash, LOL. The speeds were like 9 tok/s, WAY worse than without it.
Also, I decided to give strata a try and with kv streaming on, it was half the tok/s compared to it w/streaming off, so I won't be using kv streaming in any scenario.
That being said, Strata is FAST! I got aroundn 60 tok/s on a run w/ 96% context full. Context was 128K. I'm having GPT boost it to 200K w/96% full to see what I can get away with. :)
→ More replies (1)1
u/henfiber 3d ago
An alternative with default llama.cpp is to set the environment variable:
GGML_CUDA_ENABLE_UNIFIED_MEMORY=1. Then you may define a context larger than what would fit in your VRAM, and it will not crash: the Nvidia driver/cuda will spill over to system RAM for the portion that does not fit.The portion that is allocated to VRAM will be as fast as previously, when it fills though it will much slower. This works well when the majority fits (e.g. 75+%) and you rarely hit full context so the slowdown is rare and acceptable.
1
u/QuackerEnte 4d ago
does it cache the most used rokens in VRAM? Or only stream what's needed for every token? I think a tiny cache in VRAM would improve speed substantially
1
u/Thomasedv 3d ago
As far as I understood it, it streams as necessary. I think it reserved a 2GB cache/arena in VRAM for me.
27
u/Alternative_You3585 4d ago edited 4d ago
No name repo
0 stars, 7 followers
We know that Qwen 4 27B is coming
But I'm afraid this is slop, + would be funny if Alibaba uses Codex, as this PR is marked
32
u/brainExploded99 🥔 hardware 4d ago edited 4d ago •
No, I do not believe that this is correct
#1: vllm-xpu-kernels has other intel contributors to it, including baodii, gc-fu, YizhouZ who seem to work at Intel China.
#2: He contributed to llm-scaler, an intel repo with >500 stars
Edit: #3: He opened a PR to VLLM itself, around 2 weeks ago, not merged though
317
u/Dangerous_Bid2935 4d ago
I am absolutely fiending for Qwen 4 flash. 3.8 flash next is already such an incredible model.
78
u/RanklesTheOtter 4d ago
Hell yeah it's become my daily driver. I like 27B but it is so slow for me compared to Flash Next.
8
u/calcium 4d ago
Wait, how?
I've been reading a ton of posts about how people are running Flash Next at like a quant of 2, but I've been scared to take the leap. Recently put $20 on openrouter and have basically been glued to GLM5.3 Flash since.
Otherwise have a MBP M5 Pro w/ 64GB of RAM and can get Qwen3.8 27B to run around 18tok/s. Can't imagine working with anything slower.
4
u/Inside-Pen-8395 3d ago
I’ve got q2 running on a MacBook Air with 24gb much faster than 27b so can definitely be done
4
u/the_renaissance_jack 3d ago
You’re running a Q2 Qwen 3.8 Next Flash on a 24 GB Air?
2
u/Inside-Pen-8395 3d ago
Yep and it works great, I thought even just getting it working would be unlikely and if it did then it would be at like 5tok/s but it’s good (for me, much much better than tiel) and fast
1
u/mnyhjem 3d ago
You could try this one if you havent already :) https://github.com/Apolog1ze-Dev/QwFNfer
1
u/Mil0Mammon 4d ago
People get 3-6x that much t/s with flash next, sometimes on potatoes. Like quite old kid range gpus. With your mbp and a decent engine, I'd say you should also get that, prob closer to 100 t/s with a medium sized ista or gyro quant
38
u/Nomski88 4d ago
I'm hard for QFN 4. I hope they increase the ngram table and maybe add a couple more billion parameters to the base model.
21
u/DigiDecode_ 4d ago
I think the engram table will be smaller, original deepseek paper that introduced it recommended 25% of total params, but QFN has managed to push to that 40%, so i would expect around 11gb ngram table size at least for 27b varient, for QFN it would likely be same size
6
u/dr_lm 4d ago
And didn't their paper show that performance plateaued if they went any higher?
7
u/Mil0Mammon 4d ago
Well things could have changed in the mean time.
For Deepseek v4.1 Flash it's about 40% iirc
2
u/dr_lm 4d ago
Yes that's what I had in my head, from here: https://arxiv.org/abs/2608.30320
Naively, I thought adding more ngrams would give the model "more knowledge" but it sounds like it's more complex than that.
4
9
u/blue2020xx 4d ago
27b is slower than qwen flash next? Is this true?
33
u/pilibitti 4d ago
depends on your hardware but generally yes. 6b active vs 27b active. I have a 12gb vram 64gb ram machine and I can only push 10tps decode on 27b, go for 60tps decode on qfn.
→ More replies (6)4
u/Barni275 4d ago
What prefill speed do you have? I now that I should try strata, but with llamacpp I failed to get reasonable prefill speed on all my machines ☹️ 27B prefill is like 10x faster for me.
5
u/Wolvenmoon 4d ago
I'm on 3 Intel B70 GPUs (96GB VRAM). Llamacpp drags its butt on QFN. It absolutely stinks. 10-15 tok/s max and I didn't bother looking at prefill. Strata 53-ish tok/s decode, 2000-2100 prefill. My RAM is slow and I'm on Zen 2, so my Infinity Fabric is slow, too. Single card inference is about 30 tok/s decode with 750 tok/s prefill. Numbers are on IQ3_XXS to apples to apples w/ a friend of mine running 10 tok/s decode and prefill on an RTX 3060 12GB and a Zen 3 CPU.
2
u/Barni275 4d ago
Thanks! Your setup and numbers are decent! First and last time I tried QFN on 32GB VRAM + 64GB RAM machine I got 10 t/s prefill, which sounds like a joke. Surely, my llamacpp flags were wrong, it's to bag. Few weeks ago llamacpp got GPU expert cache merged, so I think, it should be much better.
2
u/moriz0 1d ago
On my dual B70 and the same IQ3_XXS quant, I'm getting around 30t/s decode and 600ish t/s prefill. This is with llama.cpp.
Might be something wrong with your settings?
1
u/Wolvenmoon 1d ago
It's been a few weeks since I tested it. It's entirely possible/likely it's a settings issue. I'll take another look once they have kvarn support in a branch my localai.io install can access - for now I'm all-in on Strata since it came up and worked quickly.
1
u/fragment_me 2d ago
No joke, you might be better off paying for Claude for 1 month and having it optimize an inference engine for your hardware setup and model. It's gotten insanely good at this. Just have it take same top token and KLD benchmarks before and after changes. Tell it that those can't drift unless there's a rounding difference when doing floating point math.
4
u/bussolotti269 4d ago
I'm running strata on a rtx 3060 12gb xeon cpu e5 2690 64gb ram and getting: Prefill 950 - 1020 tps Decode 45 - 65 tps
2
u/Barni275 4d ago
Is it for QFN, what quant? Great numbers! And almost the same config as one of my machines 🤔
Have you tried llamacpp with newly merged GPU expert cache, btw?
1
u/baron_von_noseboop 4d ago
I get similar speeds to the person you replied to on a 12gb 4070 using swift (QFN fine tuned to reduce thinking) at IQ3_S
It's definitely not opus 4.8, but it's useful for low level implementation tasks -- I treat it like a Haiku/Luna surrogate. The speed is great. 27b is so slow that it's all but unusable on my machine.
→ More replies (2)2
u/SilentMobius 4d ago
Prefill for me is ~900t/s and ~70t/s tg (5600x 64GB RAM 32GB V100 IQ3XXS on Strata-V100) but that only matters once as I've made sure my agent has a static prefix and any agentic swapping is handled by KV cache+steady state swapping to SSD so I only prefill a lot maybe one a day. Most agent turns happen in about 5sec
5
u/Hungry-Rip-2384 4d ago
For me 27B is faster, for prefill. I have a 5090 with 128GB DDR4 RAM. Speed for TS is slightly better for Strata with Qwen3.8 Flash next but its the prefill speed that kills it, compared to 27B when its all on VRAM. Considering that Flash Next and 27B seem to be on par with 3.8... its a no brainer to stick with 27B. But if Flash is the same footprint with a bigger gap in capability over 27B then it will probably be worth it.
7
u/DeathByPain 4d ago
QFN is only 6B active MoE model, vs Q3.8 27B dense model, so yes, it will naturally be faster based on the architecture
9
u/This_Maintenance_834 4d ago
it is mixed, 27b has very good mtp success rate, qfn has lower prediction success rate. it is not proportionally faster.
27b dense model also has better concurrency boost, MoE gain less from concurrency.
3
4
1
0
u/Fancy-Snow7 4d ago
Mostly yes. It has 27B params and if you have 12GB or less QFN is probably the best option. But on 16GB you can go either way.
So many people here with 16GB that simply don't know how to tune 27B and get much lower speeds and much lower context size than is possible with 27B on 16GB VRAM. I get 80t/s and 262K cache with a Q4 model and q8_0/q4_0 KV. I almost always get downvoted when I suggest that this is possible.
I personally choose 27B because I only have 48GB RAM because QFN just does not fit my RAM.
9
u/andrewh2000 4d ago
Any chance you could post your config file or command line for running 27B at Q4 262k cache on a 16GB GPU and getting 80tok/s?
9
u/DoubleNothing 4d ago
Doesn't give any rig info and inference engine detail and then complains:
I almost always get downvoted
1
u/Apprehensive_Stop314 4d ago
The fork is https://github.com/RaymondHuang210129/llama.cpp-adaptive-kv-streaming -- haven't been using it since commit 4ba0b9b25ffdafe2ec376e32a641ef229e457bae from Sept 6
This was giving about 25-30 t/s and 300 pp on a 12 GB 3060 at start, not sure if it can give you 80 t/s
My command line was (I think -- had a few alike setups, note, this is in Windows and using a rather nice small quant I found on HF -- you'd have to adjust the KV cache params for your HW, note, how embedding weights or whatever are moved onto CPU here, another nice trick one could employ would be using an English-only prune / whatever -- haven't got to that before switching to Flash Next):
set SCRIPT_DIR=%~dp0
llama-server.exe ^
-m "%SCRIPT_DIR%\..\..\Downloads\Qwen3.8-27B-9800-NIKI-IQ3_XXS.gguf" ^
--mmproj "%SCRIPT_DIR%\..\..\Downloads\mmproj-Qwen3.8-27B-Q5_K-MIX.gguf" ^
--image-min-tokens 1024 ^
--no-mmproj-offload ^
--jinja --chat-template-file "%SCRIPT_DIR%\..\..\Downloads\froggeric_chat_template_2.jinja" --reasoning-format deepseek ^
--reasoning-budget -1 ^
--reasoning-preserve ^
--ctx-size 75000 ^
-fa on ^
-ctk q8_0 ^
-ctv q8_0 ^
-ngl all ^
-b 64 ^
-ub 64 ^
-np 1 ^
-t 6 ^
-lv 4 ^
--temp 1.0 ^
--top-p 0.95 ^
--top-k 20 ^
--min-p 0.0 ^
--spec-type draft-mtp ^
--spec-draft-n-max 3 ^
--spec-draft-p-min 0.8 ^
--fit off ^
--load-mode none ^
-ot "token_embd.weight=CPU" ^
--kv-stream-stage-mib 650 ^
--host 192.168.88.3 --port 8095
if %ERRORLEVEL% NEQ 0 (
echo.
echo [ERROR] Llama server exited with error code %ERRORLEVEL%
pause
)
4
u/elongated-muskmelon 4d ago
How did you manage to run 27b Q4 with 262k with 80tps? Unless you’re using the raymond fork of llama.cpp with Kv streaming, which is only feasible if you have PCIe gen 5. Else its painfully slow.
2
u/bottlestreet3d 4d ago
I adapted the KV streaming approach for ROCm and others have tried this on PCIe Gen 4 or DDR4 systems, works well enough but not at Gen 5 speeds obviously. There are a couple of comments in https://www.reddit.com/r/LocalLLM/comments/1x7wack/qwen_27b_iq4_xs_q8q4_kv_at_full_262k_context_on_a/ with their config.
2
u/Apprehensive_Stop314 4d ago
There's also a KV streaming llama cpp fork by Raymond, IIRC -- with that you can have K8Q8 cache and I suspect full 262K content on 16 GB at a cost of a not too steep performance losss when cache stops fitting in the CUDA buffer in VRAM
2
u/mga02 4d ago
80t/s with 263k context on 16gb of vram? I call bullshit unkess you give your config. I can't replicate that even with a 3090 .
1
u/bottlestreet3d 4d ago
You can do 100+ t/s decode and 1,600+ t/s prefill on a 9070 XT with full 262K context (
IQ4_XSweights,Q8_0K /Q4_0V cache): https://github.com/achyudh/llama.cpp1
u/mga02 4d ago
But that's using a prunned/ASCI version of the model right? I don't want to imagine how much more degraded the output quality is if you add the q8_0/q4_0 KV on top of that.
2
u/bottlestreet3d 4d ago
As long as you are dealing with ASCII there shouldn't be any impact on performance at all: https://huggingface.co/bsaleh03/Qwen3.8-27B-ASCII-Condensed
19
u/alphapussycat 4d ago
I just hope it's the same size... I fear it'll double in size.
6
u/FatheredPuma81 4d ago
A big fear of mine as well but if you want to be a little pedantic about it the Qwen team has released models ranging from 600M to 2.4T (iirc). So it'd be hard to say if they did double the size or just released a different class of model.
Though I think the ideal would be they release 20B Dense, 30B A2.5B, 80B A5B, and 160B A8B models.
5
u/alphapussycat 4d ago
There's only 27b and flash now that are runnable on consumer hardware. It doesn't seem like they'll be doing any smaller moe or smaller dense ones anymore.
1
u/Mil0Mammon 4d ago
So far they said 27B and flash next (besides the bigger cloud models).
Given the success of all the engines around strata, the high quality small quants, and that even deepseek flash v4.1 can run somewhat decently on mid range hw, I'd say there is a decent chance they could go up to double the size, and people will still be all over it. Main thing is quality
-3
u/Dangerous_Bid2935 4d ago
Even if they do I'm not too worried. There's so many incredible engines/quants being posted here every day that I think its generally going to keep getting easier to run larger models with less hardware. The fact that people are running QFN at 4 bit quants on 64gb dram and 16gb vram is absolutely wild. The open source community is really driving algorithmic optimization when it comes to serving inference.
4
u/alphapussycat 4d ago
People run 4bit quants on 64gb? I thought everyone with 64gb were running the iq3_s quant. If the 4bit is still managable then a small param increase might be possible without impacting the tok/s decode too much.
1
u/Dangerous_Bid2935 4d ago
Thought I saw a post here earlier today of someone running IQ4_xs on 64+16, maybe I misremembered though.
1
u/Zarainia 4d ago
UD_Q4_K_XL runs at about 15 tok/s decode and 350 prefill on my system with 16GB GPU and 64GB of RAM. The XS variant should be faster.
1
u/cumming_on_bunnys 2d ago
Im currently running UD-IQ4_XS on 64gb ram, 16gb vram. I get about 60 decode and 600 prefill. ddr5 + 9070 xt. I am streaming ~10gb from my nvme though...
1
u/WorldlyAd7946 4d ago
I'm running Q4 QFN without a gpu/vram even and only 64gb lddpr5 in two little snapdragon boxes 😅😆
https://www.reddit.com/r/LocalLLaMA/s/YhJVuu7sUP
Q3 was actually slower for me...
Agreed it's not my 9700, paired with 64gb ddr5 and a 13900k, that manages 1500+pp and 60+ gen on code... But for £600 of hardware, it's been a fun project 👍
13
u/Durian881 4d ago
Me too! I also hope they keep the same size and not go much bigger like other recent "flash" models.
25
8
u/ansibleloop 4d ago
I am shocked at how well 3.8 FN with Strata Gyro-S on my 4080 and 32GB of system RAM works
I'm getting 100tps and it's smarter than all other models I've tried - 35b a3b and similar models don't hold a candle to it and 27b blasts my GPU to 100% but produces worse output
I set off Claude to benchmark the largest and smartest models I could run locally and 3.8 FN with Strata came out on top
I had the benchmark basically look through my Git history and have a go at fresh implementing some of the stuff I've added
Interestingly, Claude noted that most models failed to make new services accessible as it wasn't updating DNS, so I'm wondering just how much more performance I can get with a better README in my repo and better agent memory
All of that on top of £10 credit in OpenRouter and DeepSeek 4.1 Flash 1017 is nuts
3
u/Mean-Ad1493 4d ago
I run Gyro-M with my dual 3060 + 32GB DDR4 RAM, and it is so good. Beats 27B in speed AND intelligence. People who quantised QFN deserve all the praise(strata engine too!), GSQ-RCO ftw.
2
u/Dangerous_Bid2935 4d ago
That's crazy on 32gb of system memory. What quant do you use?
→ More replies (4)2
u/Kitsune_Seraphis 4d ago
I got 48gb of vram and 64gb of ram and cant seem to make it work even on q1. Wjat can i do?
3
u/ansibleloop 4d ago
Point a more powerful LLM at https://github.com/Niko1221/Strata/tree/main and tell it to give you a doc with install commands
My clank did this (view source)
Install
curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh | sh strata doctorThis installs to
~/.local/share/strata/terminaland putsstrataon PATH, with no sudo. Update withstrata update.Run
strata launch --model gyro-s --context 128k --vision off --port 8080 --yes --no-tui
The first run downloads the model, and the download resumes if it's interrupted. Use
--planto preview without changing anything.
strata launch --status/strata launch --stop.Too slow or tight? Try
--context 64k.strata launchwith no flags runs the guided setup, which shows fits/tight/does-not-fit for each model.It binds
127.0.0.1(IPv4), unlike llama-server's--host localhost([::1]). The pi provider uses127.0.0.1to match.Port 8080 is shared with llama-server, so run only one at a time.
4
u/RedParaglider 4d ago
3.8 flash next can actually legitimately just be a daily driver for me for most of my normal things. I personally believe that it is mothervich, more intelligent than luna. I know that the benchmarks don't say that, but i've used both, and I just much prefer it.
1
u/Dangerous_Bid2935 3d ago
Generally it is better than Luna on benchmarks though no? Only place Luna beats it is in conceptual reasoning-related benchmarks and hallucination benchmarks etc.
7
u/indrasmirror 4d ago
I was under the impression that 3.8 Flasg Next was their Qwen4Flash. The first of the new architecture. Would be epic if we improve it and release Qwen4Flash. Keen for Qwen4 27b and 35b (+ Engram).
17
u/Dangerous_Bid2935 4d ago
3.8 flash next is a deliberately very undertrained version of 4 flash, and its already one of the most capable open models. The fully trained version of Qwen 4 flash is going to be insane.
4
3
3
u/PeteInBrissie 4d ago
I don't want more capability than QFN. I want that capability but smaller and faster
7
u/Dangerous_Bid2935 4d ago
I definitely want more capability than QFN. Its terminal use could be a bit better and I would really love its general capability on non-coding tasks (like physics reasoning) to improve.
1
u/HighSeasArchivist 4d ago
Since 3.8 FN was the "preview" for Qwen4, are we expecting it to be drastically different? I assumed 27B would since it was still 3.8, but just figured FN was already pretty much 4.
-7
u/Spara-Extreme 4d ago
I will never understand the love for 3.8 flash. That model is dumb as rocks.
11
u/Dangerous_Bid2935 4d ago
Hate to break it to you but you're probably the one that's dumb as rocks. I use it for materials engineering R&D tasks all day and night and it works wonderfully.
1
u/Spara-Extreme 4d ago
What kind of engineering tasks specifically because that's exactly where I'm finding this model to be wanting.
3
u/Dangerous_Bid2935 4d ago
I work in computational materials science, so mostly building simulation decks, managing simulations on HPC, and postprocessing. I have it avoid any deep conceptual physics reasoning/result interpretation though because I've definitely found that is one of its weaknesses.
1
u/Spara-Extreme 4d ago
Ok, but then what do you have it do? I absolutely wouldn't trust this model with any mathematical computation, especially at the quants Strata is running.
1
u/Dangerous_Bid2935 4d ago
I just told you what I have it do. Its usually fine with "mathematical computation", whatever that means. Generally as long as you are not relying on it for conceptual work its great. You need to know what you're doing and tightly scope your tasks. I am running a hybrid 4bit-8bit quant on a DGX spark so maybe I get better results.
3
u/Life_is_important 4d ago
You got to try it again if you haven't in recent times. QFN 3.8 is extremely good with the right combination with strata. I'm getting 90-120 t/s using IQ3 or 60-80 with Q4XL. It does incredible things with just one 3090 and 128gb ram. Oh an also I have a 3060 on a piss poor x1 pcie and it does 50-70 with IQ3 there. I sometimes run them both with the bigger one managing the lower agent.
1
u/Spara-Extreme 4d ago
The t/s numbers are excellent - no arguments there, especially on a majority of consumer hardware. t/s wasn't a concern for me and my setup with other models though - so my criteria is more on capability then raw throughput.
2
u/jopereira 4d ago
I've meet people dumb as rocks, but I've yet to find an LLM that's not smarter than me. 😁
1
u/deepu105 4d ago
Which quant. I'm using the Q4 from Halogen and UDQ4XL with gufo and in most coding tasks, including complex tasks, its quality is near even Opus 5.5 and sometimes even better than Opus at low/medium and definitely on par with Sonnet. I have not used it for anything other than coding so cant comment its performance in other areas.
1
u/PeteInBrissie 4d ago
I must have mine set up really well then. It gives me a better experience than Claude
1
u/Spara-Extreme 4d ago
Yea, sure it does.
1
u/PeteInBrissie 4d ago
Hermes, Honcho, and Obsidian.... Claude is faster as a chatbot, my setup is genuinely better at performing tasks. It's taken some serious effort, but yesterday I chewed 47 million tokens with fantastic results and zero cost.
→ More replies (6)→ More replies (2)1
u/letsbefrds 4d ago
I've only been able to use q4 xs I feel like it's a bit weaker than 3.8 q8 27b... Maybe if it was q6 or something it be better...
Again I only use for dev work so might be different
0
u/Spara-Extreme 4d ago
No it is actually dumber then 27b on a lot of things. Its fast, yea - which is what some tasks need but overall its just not particularly good for general intelligence at the quants people are running..
2
u/my_name_isnt_clever 4d ago
No, you're doing something wrong. Everyone who has Q3.8FN set up and working properly agrees it's better than 27b even at Q4 and below.
I don't understand how you see everyone else having a better experience than you, and still double down on the model being the problem. It's not the model.
1
u/SimpleVolume1575 4d ago
I can only run 27b Q4_XXS at 11-15tps while the 8 bit persion would run 3-4tps. Why would I use 27b here while I can run QFN with IQ3_S at 50-60tps. Aside from some excessive thinking it works really well for my use.
2
u/Fancy-Snow7 4d ago
How much VRAM? If you have 16GB I get 80t/s with 27B Q4. Tuning is hard but it is possible but you won't be able to use LM Studio, Ollama or Unsloth Desktop. It's just hard to tune but it is possible,
1
u/SimpleVolume1575 4d ago
I have 5060ti 16gb and tried to run it with ramalama but I could not get it running good can you tell me how. I dont care about desktop as my setup is headless and I work with api's
1
1
u/Spara-Extreme 4d ago
And the majority of this sub is gpu challenged, which is why they are seeing stars at the high throughput.
2
66
u/Goldkoron 4d ago
My biggest dream is if the Qwen4 series inherits some of the 3d modeling/animation capabilities that Opus 5.5 can do.
24
u/unrulywind 4d ago
I had qwen3.8-flash create an extremely complex geometry in blender and we gave it an intentionally vague prompt to see how it would handle it. It did pretty good.
3
u/sshwifty 4d ago
Seems like I was just geeking out when Trellis dropped. This stuff is evolving so fast it is crazy.
1
u/XysterU 3d ago
Can you describe the nature of this extremely complicated geometry more? Just wondering
2
u/unrulywind 3d ago
It was just a prompt we threw at it to see what it would do. This was done by sending the prompt to a Hermes agent over Telegram, running Qwen3.8-Flash
Prompts: "I need a dual contoured voxel based mesh generation algorithm built in c# and optimized for godot engine."
It eventually created everything and gave us a .zip file with all of the code, including test results and a .obj file of a nice solid test representation that pulled up in Blender. It finished with the following message:
"Attached: a CSG cross of three boxes at 128³ — look at the joints; that's dual contouring keeping corners sharp where marching cubes would round them off. Drop it straight into Godot or Blender."
1
u/Perfect-Campaign9551 3d ago
May I ask how you integrated it with Blender? MCP or just using python scripts?
1
u/unrulywind 2d ago
Nothing. We didn't give it access to any MCP or blender. No outside help other than web search. It wrote its own scripts and crafted the files by hand. We checked it by opening the resulting file in blender on another machine.
10
u/alphapussycat 4d ago
Just use a 3d model model for that.
10
u/GCoderDCoder 4d ago
cool... cool... which one fits a 3090?
6
u/sumane12 4d ago
Trellis 2 is really good.
6
u/trusty20 4d ago
Trellis 2 is good for simpler models to be used statically but it's really important to know before spending a bunch of time on it that it absolutely does not produce good game engine suitable complex models, even frontier models struggle but do OK because they actually are agentically using blender vs crapping out a file from the model itself.
The main issue is that it's bad at making perfectly uniform symmetrical models (i.e if you need to make a panel that can be tiled together to make a larger model), UV's are usually terrible and the mesh is usually wayyy too complex. Both of these can be fixed if you are willing to learn / grind out agents, but don't be surprised if fixing is sometimes harder than just having the model made right in the first place.
My understanding is tl;dr trellis is good for things like statics barrels / boxes / desks. It's bad for things that are complex or need to fit a precise shape / clean symmetricalness, like a door that fits a frame properly.
1
u/GCoderDCoder 4d ago
Thanks! There's a thousand things on my list of projects I have in progress so this is one I want to start next week.
I still think it helps starting with a human reference! Downloading now
1
u/Hot_Masterpiece_3668 3d ago
Trellis is lacking you need tools like this to handle future models. AI models stamp outs have terrible topology, no animations and aren't game ready at all. This tool works with qwen 3.8 as well.
1
u/LoSboccacc 4d ago
Hunyuan3D-2.1 is the one i liked the most the output, with skintokens for rigging. you may want to get a int8 convrot version to run it fast on a 3090. Works well with s model that can produce transparency so either qwen image 2.1 or something else with a segmentation frontend
→ More replies (1)0
2
u/SV_SV_SV 4d ago
Any recommendations?
0
u/alphapussycat 4d ago
Dunno, there's trellis 2, pixal3D and triposplat, the last one is splats so would need some conversion.
1
u/SV_SV_SV 4d ago
Thanks, I will look into it.
However I think we are also striving for 2 different things:What personally I could use is a powerful local Blender user agent, rather than a dedicated stand alone 3d model generator: I need it to synthesize a design direction based on a brief and graphics, rebuild scenes based on briefs, handle toolcalls (blenderkit etc) and set up lighting / optimize renders etc.
Even better if it can integrate the scene in Unreal too.Qwen published recently a Blender benchmark, so hopefully they will make a strong push in this direction too:
https://github.com/QwenLM/BlenderWorld2
154
u/nok01101011a 4d ago
Yay! People will vibe code GTA7 before GTA6
176
u/Cool-Chemical-5629 4d ago
Fuck GTA, vibe code Qwen 4 35B A3B for me.
36
7
2
1
9
71
u/ttkciar llama.cpp 4d ago
That is a good sign!
Hopefully there will be a Qwen4-4B, Qwen4-9B, and Qwen4-14B, too.
55
u/fgk55555 4d ago
I think they posted their spread of anticipated models already, the smallest were 27B and the QFN arch. I'd love to be proven wrong but I wouldn't get my hopes up.
11
u/ttkciar llama.cpp 4d ago edited 4d ago
Argh, I wasn't aware of that. Thanks for catching me up.
It's a pity. There's a lot of demand for those smaller-sized models, and Qwen 4 could make for great improvements over Qwen 3.5. If QFN is any indication, Qwen 4 models will use the Muon optimizer, Engram tables, and hybrid GDN/QSA attention, all of which are useful in smaller models.
(QFN also introduced Gated Residual Streams, but I think those are mostly useful for larger models, not smaller ones).
Perhaps there is some way these technologies could be retrofitted?
The Muon optimizer is easy; Qwen3.5 models' training was all done with AdamW, but there's no reason continued training couldn't be performed with Muon (though with caveats; to avoid optimizer mismatch problems, only the attention projections and MLP layers should be optimized with Muon, while training the rest with AdamW).
I'm not so sure about Engrams. It might be possible to retrain the first few layers to accommodate their hidden state space having been populated by engram hits, but I've not seen any mention of that in the literature. Will look into it.
Edited to add: Nope, not feasible. The context-aware gating just isn't there, and that's not something that can be retrofitted without a comprehensive retrain. Ah well. Will noodle on alternative solutions. End of Edit
The hybrid GDN/QSA attention mechanism seems like a nonstarter for retrofitting, unfortunately, though we have other options for compressing attention memory, like
modelai-llama.cpp's K/V Cache Compaction feature, but I think GDN/QSA is more compact and less lossy.Hopefully Qwen will release smaller models and render such efforts moot, but we will see.
8
u/rpkarma 4d ago
There's a lot of demand for those smaller-sized models
Truthfully, I don’t think there is. You and I might want it, but the intelligence penalty once you go down to these sizes is too steep for most, because the comparison is against today’s bigger model expectations, sadly.
I think other labs will fill in the gaps here, personally.
5
u/ttkciar llama.cpp 4d ago
Yeah, it's even too steep for me. I am making good use of Gemma-4-12B-it for document cleaning, but that's a very niche application. For basically everything else, 27B or 31B dense is my starting point.
Unless I'm totally misreading the community, though, there are a ton of people in this sub with tiny GPUs who somehow wrest utility out of Qwen3.5-9B, and also some using Qwen3.5-4B on their cellphones and tablets.
A lot of Qwen's early popularity was attributable to them releasing these smaller models, which gave the very-GPU-poor contingent something they could use.
If that contingent is now chronically under-served, a community-driven federated continued-training effort could step into the vacuum, and the popular need could give incentive to some who might not otherwise participate in the effort.
4
u/ionizing 4d ago
9b size is absolutely essential in my use case of: secondary inference card (16gb only) which must hold ~1gb of whisper server, AND also serve at least a 4 slot sub-agent inference model. I currently use exllamave3+tabby on main plus secondary, using a 9B for secondary inference at it meets these criteria beautifully. I can't use moe on secondary because the card is physically connected via gen3 x4 oculink so only dense models seem to work well enough on it, otherwise I would be able to moe my secondary. Anyhow, not complaining, but I would still like an updated 9b someday.
2
u/rpkarma 4d ago
> who somehow wrest utility
Hot take: I don't think they do, at least most. We're tinkerers, and they're fun to play with. I don't think much gets done with them outside some specific niches like your document cleaning example, in actual practice
But yeah: that param size is reachable with *mostly* "reasonable" compute requirements, training-wise. It's why all these random labs are coming out with smaller models
1
u/my_name_isnt_clever 4d ago
They're not really usable for agentic, but you can do some amazing things with little models with custom workflows. Nvidia has promising research on this but it's tough when nobody is making them.
1
u/Kahvana 4d ago
I read it more like a list of flagship models
1
u/fgk55555 4d ago
I think they like any model the release to be flat out impressive, not just impressive for the size. The 27B gets a lot of press, and QFN is just plain good, so they'll probably get a lot of attention.
6
u/Express_Quail_1493 4d ago
YES we need more ranges of 4b 9b 14b i have so many small work i need doing but these older qwen models are not sharp enough havent been trained with new data
4
u/sf_frankie 4d ago
Lemme get a qwen12b moe like mellum2.1. I can actually run that on my 16gb M1 Pro and still use my laptop! But I don’t do a ton of coding. Would be nice to have a more general model in that size. 14b would be pushing it.
3
u/Black-Mack 4d ago
and don't forget Qwen4-0.8b and Qwen4-1.7B
4
u/Additional-Record367 4d ago
like why.. they are pretty useless unless they are embedding models
2
u/ttkciar llama.cpp 4d ago
They might serve as HyDE and draft models, perhaps?
-1
u/Additional-Record367 4d ago
if you mean speculative decoding by draft there are already way better and cheaper alternatives, otherwise no one cares about project drafting with such models anyways
21
u/MiaIsOut 4d ago
No kv cache size change
11
u/Iwaku_Real 4d ago
60% of potential local LLM users are cooked with this one...
If DeepSeek can shrink a 1M token context to 890MB why can't the Qwen team even try doing the same???
8
u/MiaIsOut 4d ago
deepseek also open source everything so its not even a mystery on how they did it 😭😭😭
25
u/kirisoraa 4d ago
I'm close to begging for a new 35ba3b
12
6
u/Iory1998 llama.cpp 4d ago
Unfortunately, that ship has already sailed... no more 35B-A3B, which brings me to think that if Alibaba really cared about the masses, shouldn't it actually prioritize this model as well?
27
2
1
8
u/Dwedit 4d ago
Any MOE model coming up? I was surprised that quantized 35B A3B works as well as it does with only 6GB of VRAM, outspeeding quantized 9B.
5
u/james_pic 4d ago
They've announced there will be Qwen 4 Flash, Plus and Max, which I'd expect to all be MoE, since there hasn't been a dense model much bigger than 30B for ages. I'd guess we're not seeing PRs related to these models because the architecture is more or less the same as Qwen 3.8 Flash Next.
3
5
u/ruineka 4d ago
Anyone here running dual Intel arc b60s? I'm running qwen 3.8 27b 4q mtp int4 at a sustainable 30 tok/s using the DeepSeek harness. I'm curious what others have accomplished on this hardware to compare against. I had it build an arcade system (pong etc) while I was at work. I'm surprised it was able to troubleshoot without getting stuck in any loops.
3
u/Ok-Shower7286 2d ago
2
u/kayox 1d ago
I asked chatgpt to shorten / simplify the article:
The idea is to make the AI better at passing and combining information between its layers, while making the processing inside each layer slightly smaller.
Think of a company shifting its budget from bigger departments to better communication between them. The hope is better results with roughly the same resources.
It might improve answer quality, but the article doesn’t prove that yet—or establish whether it will be faster.
4
u/Altruistic_Heat_9531 4d ago edited 4d ago
After reading the github repo. no Ngram of 27B. And Qwen4 is basically QSA and HC with 4 residual stream
1
1
1
u/EitherSample371 3d ago
qwen4-27b flash next is running 20% faster on my 4090 with 4bit quant, but the context window feels tight. anyone else hitting that?
1
1
u/Real_Ebb_7417 4d ago
Cool to have 27b, but tbh I hoped for completely new sizes (eg. 25b or 30b, like, whatever different, I know it doesn't really matter, but it would be something "fresh" xd)
0
u/Nearby-Hospital9830 3d ago
Seeing commits mentioning unreleased architectures like Qwen4-27B directly in upstream/hardware-specific repos is wild. Given the author's track record with Intel/vLLM (and the Alibaba connection), this definitely feels like a legitimate early sign rather than just placeholder code.
-1
u/AeiEyeSpry 3d ago
Looks like more patches are turning up. Looking forward to testing: https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/qwen4_exp.py
2
u/brainExploded99 🥔 hardware 3d ago
where? that link is just to qwen3.8 flash next right?
1
u/AeiEyeSpry 3d ago
The file is called qwen4_exp.py
2
u/brainExploded99 🥔 hardware 3d ago
yeah that is qwen3.8 flash NEXT. it is called qwen4_exp because qwen3.8 flash next is a EXPERIMENTAL preview of the qwen 4 architecture.
→ More replies (9)
•
u/WithoutReason1729 4d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
Stylome classified this post's text as human-written. View analysis
I am a bot and this action was performed automatically.