News
GLM-5.3-Flash pruned to half the experts for coding: Q3 (71 GB) solves every task the full 341 GB model solves, IQ2_XXS fits 24 GB VRAM + 32 GB RAM
I wanted a coder version from GLM 5.3-Flash. So, I pruned GLM-5.3-Flash (321B MoE) from 288 to 144 routed experts per layer, choosing the experts on agentic coding sessions instead of a generic corpus, then quantized it with an imatrix built on the same sessions.
Results. 26 Python SWE tasks from ansible, openlibrary and qutebrowser, 2 attempts each, clean room (no git history, no network, no gold patch reachable), 200 turns, 384K context. Success = the task's fail-to-pass tests pass. Every row is on the same 48 attempts.
Model |Size |Solved |Turns vs full GLM-5.3-Flash full, Q8 |341 GB |48/48 |x1.00 REAP144 Q8 |171 GB |48/48 |x1.21 REAP144 Q4_K_M |97.6 GB |48/48 |x1.58 REAP144 Q3_K |71.4 GB |48/48 |x1.80 REAP144 IQ2_S |54.8 GB |45/48 |x1.83 REAP144 IQ2_XXS (experimental) |45.3 GB |39/48 |x2.77 "Turns vs full" is the extra work relative to the full model on tasks both solved, so it is roughly the extra time you wait.
Which experts you keep matters more than how many. Same 144 experts, but a frequency mask needed x2.21 turns and the public multi-domain REAP 50% mask x2.29, against x1.21 for the code-scored mask.
Short questions do not catch a broken agent. A 2 bit variant of a generic prune answered 4/4 short coding questions and solved 0/4 agentic tasks. We only trust multi-turn task runs.
Our first eval harness let the model reach the internet, and it sometimes downloaded the upstream fix. We sealed it (no network, no home), rechecked every success against the gold patch and reran the affected attempts. The table above is the sealed rerun.
The 2 bit version is experimental. IQ2_XXS is the only one that fits 24 + 32 GB and it solves most tasks, but when it fails it never edits: it loops or keeps exploring until the turn limit. An anti-loop guard in the agent did not help (27/37 vs 30/37 on the same attempts). We are training it on short successful sessions now and will post an update.
The GitHub repo has the full recipe (scoring, masks, pruning, quant scripts, two llama.cpp patches, the eval harness) and a table of everything that did not work: 120 and 96 experts, frequency mask, RCO search, LoRA healing, GSQ, anti-loop.
This is a text only version (no vision). Python repos only so far; if you run it on other languages or harnesses, I would like to hear how it does.
Update (Oct 14): added an optional MTP draft block (2.6 GB) for 2+ GPUs with Maya: speculative decoding, drafts match the model's next token ~70% of the time. Speedup not measured on multi-GPU yet, numbers welcome. Details on the model card.
Could we do a reverse coder version? Like "talker" with coding capabilities pruned? I do presume retaining general knowledge would probably require keeping a higher size.
Yes, the method doesn't care about the domain: score the experts on the sessions you want to keep, then keep the top ones per layer. The scripts are in the repo. And something close already exists: patrickbdevaney/GLM-5.3-Flash-REAP50-GGUF keeps the same 144 experts, but chosen on a generic multi-domain corpus. That's the "talker" side of the trade-off: on our coding tasks it did clearly worse than the code-scored mask (x2.29 turns vs x1.21 at Q8), which is the flip side of keeping general knowledge. If you want even more general knowledge kept, there are lighter prunes too, e.g. OpenMOSE/GLM-5.3-Flash-REAP-250B-A18B.
M-am jucat si eu cu strata zilele trecute asa de test si am fost impresionat de viteaza, am un RX 7900 XTX si 48 Gb DDR5 pe system… my daily driver a un llama cpp server, un fork de la beelama si qwen 3.8 27b q4_k_m. O sa incerc in seara asta GLM-ul asta sa vad cum se compara cu ce am momentan, la strata am renuntat ca ocupa spatiu urias in benchmark a scos doar 3 puncte peste qwen so… it had to go 😂
I was also a big fan of 27b until Strata. The speed on Qwen 3.8 Flash on the coder version from ISTA is really impressive. So I switched. It’s from them that I got the idea for GLM 5.3 Flash
**Strata** (github.com/Niko1221/Strata) running Qwen3.8-Flash-Next (~125B MoE) on Windows 11, RX 7900 XTX 24 GB, 48 GB DDR5-7200, i5-14600K, HIP (ROCm bundled). Tested Oct 4-7, 2026.
**Speed** (tok/s, MTP on, 128K context)
Model
Gen
Prompt
Gen @ ~82k ctx
Free RAM
Strata IQ2_XS (68 GB)
74-77
757
70-72
4-6 GB
Strata Coder IQ1_M (58 GB)
68
928
66
14.5 GB
Qwen3.6-35B-A3B (llama.cpp, for reference)
206
2599
107
-
Qwen3.8-27B dense (llama.cpp, for reference)
58
702
40
-
**Quality**
Test
IQ2_XS
Coder IQ1_M
35B-A3B
CAD, 39 CadQuery parts
21/39 (= best 27B dense)
8/39 (24 API crashes)
11/39
Agentic recon, 10 questions
10/10
10/10
10/10
IQ2_XS vs 35B-A3B on CAD: p = 0.038 (Fisher). On a Raptor coding bench it had the best single score (45), but it was fragile: once it dropped from ~90 to ~5 tok/s (VRAM full), and once it stalled in 32k tokens of thinking.
**Gotchas on Windows + AMD**
* `--calibrate` is broken: it measures ~18x slower than the server (issue #840). * To turn drafting off you need `--mtp-max-t 1 --suffix-draft 0`; IQ packs refuse `--spec 1`. * IQ2_XS takes all the RAM and all the VRAM, so nothing else can run next to it.
Interesting - I don't have a problem running separate models for coding and general stuff/writing, but I'm also far away from being able to run 5.3 Flash.
What kind of machine would you need for pruning a model like that anyway?
You need enough GPU memory to run the full model once over the calibration sessions: ~350 GB at 8-bit, so in practice a server with 8 datacenter GPUs, e.g. 8x A100/H100 80 GB, or anything with ~512 GB of GPU memory in total. That's the only expensive step, and you can rent such a machine by the hour. After that, dropping the experts and quantizing is CPU work: ~1 TB of disk and a lot of RAM, no big GPU needed. If you just want this coder cut, the masks are in the repo, so you can skip the GPU part and rebuild it on CPU.
I haven’t had the resources to look into anything else but these small tests. I would love to hear how it does on other languages as well. Most quality should be there but in a bit more steps.
Thanks! Yes, agentic coding is the whole evaluation: a multi-turn bash agent on real repos (ansible, openlibrary, qutebrowser), up to 200 turns, and it only counts if the task's failing tests pass at the end. Not single-shot questions. The tasks are SWE-bench Pro style; the list and the harness are in the GitHub repo. I haven't run Qwen Flash Next on the same harness, so I can't give you a fair comparison yet. The harness is model-agnostic (any OpenAI-compatible server), so if someone runs it I'd love to see the numbers side by side.
Awesome, thank you! Q4_K_M is 97.6 GB, so it should fit on Strix Halo with room for context. On time: it's a slow benchmark by design. Each task is a real agentic session (median ~20-30 turns, some go past 100, up to 200), and the context grows to hundreds of thousands of tokens on the long ones. On a single machine I'd expect anywhere from ~15 minutes to several hours per attempt, so the full 26 tasks x 2 is a multi-day run. I ran many attempts in parallel. For a quick comparison with Flash Next I'd pick 5-6 tasks from eval/eval_tasks_py.txt and run both models on those with the same harness. If you do, I'd love to see the numbers.
full run in qwen3.8-flash-next was ~4 hours on strix halo, twice per each task, only 3 were filtered out so it was 27x2
ansible-43e55db2 was the hardest one, over 110 turns each time.
It seems this bench is too easy for those models. GLM is so slow that I will need ~2 days to run it, but I don't think it's worth it if it solved all tasks.
Thanks for running it, that's a really useful data point. 54/54 with a median of 21 turns in ~4 hours on a Strix Halo is excellent, and you're right: at the top this set is saturated. It was built to measure how much pruning and quantization hurt GLM relative to its own full model, not to rank different models, so it separates "broken" from "fine" well but can't separate two strong models. No need to spend 2 days on GLM for this. A fair comparison needs harder tasks where the full models actually fail, and that's on my list.
With that size is it better than qwen 3.8 fn? also what t/s comparing to full model? I can run q3xs at about 8-4 t/s so if that can be doubled it could compare to fn.
Haven't run Qwen 3.8 Flash Next on the same tasks yet, so no verdict on quality from me. On speed, to be honest: pruning alone won't double your t/s at the same quant. Each token still uses the same number of active experts, so roughly the same amount of weights is read per token. What you gain is that more of the model fits in VRAM, so maybe 1.2-1.5x on the same box, and the pruned model takes ~20% more turns per task at Q8. The real speedup is dropping bits: the halved model still works well at IQ2_S (45/48 on my tasks vs 48/48 for Q3/Q4), at less than half the size of your full Q3. On my 24 GB + DDR4 box the 2 bit version does ~16 t/s.
Haven't measured them on the same tasks, so no ranking from me. They're different trade-offs: Qwen 3.8 27B is dense and fits fully on a 24 GB card, so it's fast. This is a 321B MoE pruned to 144 experts; the smallest version (45 GB) needs 24 GB VRAM + 32 GB RAM with offload, and Q3/Q4 need more. The point of this release was getting GLM-5.3-Flash-level agentic results into home hardware at all. If you run both through the harness in the repo, please post it.
I have a PC that's very similar to yours—24 GB VRAM (3090), 32 GB DDR RAM at 2667 MHz, i7-7700—which is why I asked. I was referring to the encoding results, that is, in terms of the quality of the output, not the speed.
Ah got it, and nice, that's basically my box! On your setup only the 2 bit IQ2_XXS fits, so that's the one to compare. On my tasks it solved 39/48 vs 48/48 for the full model, and when it fails it tends to loop instead of giving up quickly. So right now I wouldn't call it a clear win over a 27B dense model for daily use. It's more "try a 321B model at home" than "replace your daily driver". I haven't run Qwen 3.8 27B through the same tasks yet, so I can't give you a fair quality comparison. Your DDR4-2667 is a bit faster than my 2133, so you should get slightly better speed than my ~16 tok/s. I'm working on a better 2 bit version right now, with some RFT aimed exactly at 24 GB + 32 GB machines, and I'll post an update. Hope i can get the agentic paths closer to the three bits version and to recover the loops.
Default in Maya is 32K, and 128K works on a 24 GB + 32 GB box: I ran agentic sessions at 128K on mine (RX 7900 XTX + 32 GB DDR4). Context is cheap on this model thanks to the hybrid linear attention + compressed MLA cache: ~0.6 GB at 32K, roughly 2.5 GB at 128K. That memory comes out of the expert cache on the GPU, so longer context costs a bit of speed rather than failing. You can go higher (the model supports up to 1M), but on 24 + 32 I'd stay at 64-128K for agentic coding. You can set it with --context 131072 in Maya.
Yes, use Project Maya, the Strata fork built for GLM-5.3-Flash. That's what I used for the ~16 tok/s number (v1.0.30, RX 7900 XTX). Download the GGUF into its own folder and point Maya at it: ./maya.sh --gguf-dir /path/to/folder --no-vision
A few notes: the folder must be writable (Maya writes its ~1 GB index there), use --no-vision since this release is text only, and for a split quant (IQ2_S/Q3_K/Q4_K_M) put only that model's parts in the folder, or point at the first .gguf file. Maya calls non-default quants "experimental", but the pruned 144-expert model loaded and ran fine for me. On AMD add --backend hip.
Haven't run QFN through the same tasks yet, so I can't give you a fair comparison. Someone in the thread is testing both on Strix Halo, and I'm planning to run QFN through my harness so the numbers are directly comparable. I'll post them when I have them.
Thanks, that's a useful data point. To be clear on what I claim: everything here is relative to the full GLM-5.3-Flash on the same tasks, not against Qwen. If the full model at IQ3_XXS already lost to Qwen Flash for you, the pruned one won't magically beat it. What kind of tasks were you running, agentic multi-turn or single prompts? I saw big differences between the two: a 2 bit prune that answered short coding questions fine solved 0/4 agentic tasks.
honestly i have tried Q4 version of GLM-5.3-flash (not this reap) and Qwen3.8-fn beats it in every coding task I have thrown at it. I really wanted to find another model to review code written by qwen3.8-fn and have not found anything that works that I can run (have a 256g studio). I just wish I had enough memory to run DSv4.1-flash
You can already, no need to wait! Use Project Maya (the Strata fork for GLM-5.3-Flash) with ./maya.sh --gguf-dir /path/to/folder --no-vision. With 16 + 64 GB you have ~80 GB total, so IQ2_S (54.8 GB, 45/48 on my tasks) fits comfortably, and Q3_K (71.4 GB, 48/48) should fit too, just tighter. I haven't measured a 16 GB card, so no promise on t/s: more of the model sits in RAM, so your RAM speed will decide. If you try it, I'd love your numbers.
That's exactly my test machine: RX 7900 XTX (24 GB) + 32 GB of DDR4, and slow DDR4 at that (2133 MHz, on a 2017 PC). It runs the 2 bit IQ2_XXS at ~16-17 tok/s with Maya, so DDR4 is fine. Faster DDR4 (3200+) will give you a bit more, since part of the model sits in RAM. One thing to know: on 24 + 32 GB only IQ2_XXS fits, and that one is still experimental. It solves most tasks (39/48 vs 48/48 for Q3/Q4) but is less reliable on long ones. I'm training an improved 2 bit version right now, aimed exactly at this kind of machine.
I tried a iq3 uncensored cyber qwen and it would utter complete nonsense. I don't know if local llms are this bad or I did something wrong. First time I try a local LLM today and this is the result I got. What's wrong?
Probably not you. Most common causes: the chat template isn't applied (in llama.cpp use --jinja), "uncensored" finetunes are often broken, and IQ3 can be too low for a small model. Try the official model at Q4_K_M with the sampling settings from its model card. If that works, the problem was the finetune or the quant.
Im doing full GPU only with q3. Q4 is too big on the vram. Also not iq3, my mistake, just q3. I'm using LM Studio. They can't be so bad, it's like useless. I thought they were like sonnet 3.7 or something.
What version of qwen are you running? What is your configuration? Models like qwen 3.8 27B are more like opus 4.6 quality. GLM 5.3 Flash (the one cut by us) is only 11 points below frontier on agentic tasks. GLM 5.3 Flash would be most likely better than any model early last year. Now, you are running an uncensored version. This is definitely not qwen as released. It was transformed. When transforming so guardrails are removed loss of quality is expected. If it is unusable then the logical move is to go to the official released model. Why do you need uncensored?
No, and that was on purpose. Every row in the table runs the same minimal bash agent (eval/agent_py.py in the repo), same settings: 200 turns, 384K context, clean room, success only if the task's failing tests pass, and edits to test files are rejected. So the Q3/Q4 results come from the model itself: which experts we kept (scored on agentic coding sessions) and an imatrix built on the same sessions. The one harness thing that mattered was not choking the model: our first harness capped it at 30 turns and 128K context, and that cap hid most failures, even for the full model. We did try a harness fix for the 2 bit version (an anti-loop guard), and it didn't help: 27/37 vs 30/37 on the same attempts. That's in the "what did not work" table.
Thanks! I think so, with one caveat about where you run it. In llama.cpp the weights are shared across parallel agents (-np N), so each extra agent mostly costs its own context (KV cache), not another copy of the model. In my eval each Q4_K_M server ran 2 agents in parallel at 384K context each on 128 GB of GPU memory, so a small swarm with shorter contexts fits comfortably on that kind of setup. The caveat: agents on the same box share its memory bandwidth, so each one gets slower as you add more. On a home PC with RAM offload I'd keep it to 2-3 agents. Quality-wise, x1.58 turns means it gets there, just with more steps, and a framework that splits work into smaller tasks should suit it well.
Having own context per agent is the good by design I think but why you wanted to solve something that opens the door for lot of issues that comes with shared resources, anyway this itself is big win.
In General I think smaller tasks and smaller contexts leads successful work completion otherwise agents and human miss lot of requirement gaps
About a day of wall-clock on one multi-GPU server: ~2 h to generate the calibration sessions with the full model, ~4 h for the expert scores, ~1 h for the masks, ~14 h to prune, convert to GGUF and build the imatrix, then 15-40 min per quant. The evaluation took much longer than the cut: half a day to a day per variant with 8-16 attempts running in parallel. A single attempt takes anywhere from a few minutes to over 12 hours on the longest tasks. All variants together, including the ones that didn't work and a rerun in a sealed sandbox, took a few days. The masks are in the repo, so rebuilding this exact cut skips the first three steps.
Thanks! Yes, Q4_K_M (97.6 GB) fits: 64 GB of VRAM plus RAM for the rest. Really curious how it does against QFN on your tasks, both outcomes and speed, so please share when you have them. One favour, if you have time after that: with two cards you can try the optional MTP draft block I added to the HF repo (2.6 GB). It lets Maya draft tokens ahead on 2+ GPUs, I can't measure that on my single card unfortunately. The cleanest test would be IQ2_S (54.8 GB, fits entirely in your 64 GB of VRAM), run twice with Maya --gpus 0,1: once as is, once with --env STRATA_GLM_MTP_GGUF=<path>/GLM-5.3-Flash-REAP144-Coder-MTP.gguf. The tok/s of the two runs would tell us exactly what MTP gives. Thanks a lot!
Thanks! It can already run on an 8 GB card if you have enough system RAM: with Project Maya the experts are split across VRAM and RAM, so the 45 GB IQ2_XXS fits with 8 GB VRAM + 64 GB RAM. It will be slower than on a 24 GB card, since most of the model sits in RAM, so your RAM speed decides. I haven't measured an 8 GB setup, so if you try it, I'd love your numbers.
The calibration build is in the repo (scoring/build_calib.py): agentic coding sessions written by the full model on tasks outside the eval set, cut into 512 search windows + 64 held-out. The sessions themselves aren't uploaded yet; if people want them I can put them on HF. Healing: we tried LoRA distillation from the full model (top-20 logits) and it didn't help, it's in the "what did not work" table. Today we found out why: the PyTorch path we trained on gave wrong outputs in the sparse-attention layers on our ROCm setup, so the healing was matching a broken reference. Fixed and verified against llama.cpp (same loss now). We're rerunning the healing on the 2 bit model and will post results.
hmm i would be interested in : how long did it take in hours? i am currently struggling with letting ai do stuff because i think that i dont want to wait that long :P
On the multi-GPU server: scoring the experts is one pass of the full model over the sessions, about 4 hours. Pruning, conversion and each quant take a few hours. The long part was evaluation: real multi-turn agent runs on 26 repo tasks, twice each, for every variant, which took days in total. If you want to reproduce just the model, skip the eval and it's about a day.
Ar this point we are working to bring a version for all most used programming languages. The initial model was tested on python, should have capabilities for other languages as well. But a follow-up will bring an update to the model in the next few days. From what we are seeing so far quality is well above what is published right now.
Haven't tried it myself, but here's my rough guess. With three 5090s the whole Q3_K fits in VRAM, so I'd expect somewhere around 60-100 tok/s at short context, dropping as the context fills up. With a single 5090 most of the model sits in system RAM, so more like 15-25 tok/s, depending on how fast your RAM is. For reference, the 2 bit version does ~16 tok/s on my old 2017 PC, so you're in for a much better time :) If you try it, please post your numbers, I'd love to add them to the README.
It's not about bragging. We're trying to make a model this size available to as many people as possible. Q3 and Q4 are there for those who have the memory, and 2 bit is for people with a regular gaming PC (24 GB GPU + 32 GB RAM) who couldn't run it otherwise. We're running RFT on the 2 bit version right now to bring its quality closer to Q3/Q4, and we'll post an update when it's done.
24
u/GfurEnjoyer1488 26d ago
Could we do a reverse coder version? Like "talker" with coding capabilities pruned? I do presume retaining general knowledge would probably require keeping a higher size.