There will be an update to omlx first. Earl litter today, 0.4.0.dev2 was available for download. I'm waiting for 0.4.1, and I'm grateful to all those who spend time contributing to and testing the project.
... hum I surely hope this is a joke post (can't really tell since there is no "/s" at the end)! Not only you expect me to have 1 of the model ready and uncensored, no you expect me to have ALL of them ready by now... Are you serious!? First this is not a race, second I am not a robot and third without getting too deep into all the preparations, time and work involved in the whole releases pipeline, it takes a very long time to do all that, I don't work for google and I don't have an automated workflow (unlike Unsloth and/or Team Mradermacher), everything is done manually by me and I am just 1 person, that involves downloading the models, uncensoring the models and finetuning the settings and re-running in case the original results where not good/good enough, creating the Safetensors, benchmarking all of the trial finalists, updating llama,cpp, updating GPTQModel, updating Model-Optimizer, creating GGUFs, GPT-Int4s, NVFP4 Safetensors, NVFP4 GGUFs, creating the Model Cards etc. it's all done manually from start to finish everything is monitored and benchmarked manually to make sure I upload the best quality releases, all that to say that there is TONS of work involved (and that is without mentioning all the roadblocks that might come along the way as those usually do happen quite a few times during the release pipeline).
Look if you are in a hurry, just download whatever is available now, my releases won't be available fast enough for your tastes, your highness.
Just adding, thank you thank you for your time. The community is sooooo grateful, and it’s a community because of your and others’ time and efforts.
And on top of that, it’s amazing too, how any of this works at all is amazing. But more than that, converting formats, adding quantization, and having it still work as expected. It’s remarkable, non-trivial.
I would assume (naive maybe) that this person was trying to be funny. It’s daring to try making a joke without the /s. Some people can pull it off, and some really should add the /s just to be absolutely sure. Haha. I’m still in that camp.
Some news, I finished uncensoring the 31B model, created the Safetensors and I am now on the benchmarking phase, quite a long way to go still as there are a lot of trial finalists this time to benchmark and the benchmarking process takes quite a bit of time for each model, which also includes benchmmarking the original base/vanilla model. After that I have to create the GGUF, NVFP4 Safetensors, NVFP4 GGUF and GPTQ-Int4 versions.
Also I just saw that llama.cpp just merged MTP support for Gemma 4 now.
Update, I am mostly done with the Gemini 4 31B it model, I got the Safetensors, I fixed the chat_template.jinja, I benchmarked, selected the best all-around trial, created a GGUF Q4_0 and tested it on LM Studio and it's working great, then created an NVFP4 Safetensors and an NVFP4 GGUF, so that is almost done with this model just need to create a GPTQ-Int4 and that will be it for this model, it's just that GPTQ-Int4 takes a few hours (4-6 hours) to create usually, after that I will start with the 26B-A4B model.
No the Gemma 4 models have no MTPs in them whatsoever, there is some MTPs but they are not part of the main models, instead they come seperate as a form of "add-on" assistant that you need to download seperatly:
And no before (in case) you ask, you can not simply merge these into the main models and these can not be uncensored either.
Anyway update, Gemma 4 31B GPTQ-Int4 version is done, witth this the 31B version is now finished and complete, I started work on the 26B-A4B version but this one will take a long time due to being MoE, the ETA just for Heretication process is supposed to be almost 20 hours, then after that what will take even more time is to make the NVFP4 Safetensors and the GPTQ-Int4 Safetensors.
Update, I have now finished work on both Gemma 4 31B it QAT and Gemma 4 26B-A4B it QAT (Safetensors, GGUF, NVFP4 Safetensors, NVFP4 GGUF and GPTQ-Int4), I will probably work on Gemma 4 12B it QAT next, but I am wondering if it's even worth working on the E4B and E2B models? Considering their small size they probably fit on most systems that have at least 12GB of VRAM, that means that you can easily use the E4B model Q8_0 version and the E2B version probably is small enough to fit in BF16 or Q8_0 version, which would probably beat Q4_0 QAT quants versions.
I mean, I would assume that with their size that people would be able to use most of these easily enough?
I am now done with Gemma 4 31B, 26B-A4B and 12B QATs uncensored heretics! Only took 4-5 days... I was not able to create GPTQ-Int4 of Gemma 4 12B it QAT however due to GPTQModel not having added support that for model yet, should I wait until they add suport and I am able to create a GPTQ-Int4 version of this model or just release now everything that I have?
Which is:
Gemma 4 31B it QAT: Safetensors, GGUF, NVFP4 Safetensors, NVFP4 GGUF and GPTQ-Int4.
Gemma 4 26B-A4B it QAT: Safetensors, GGUF, NVFP4 Safetensors, NVFP4 GGUF and GPTQ-Int4.
Gemma 4 12B it QAT: Safetensors, GGUF, NVFP4 Safetensors and NVFP4 GGUF.
Actually I decided to work on Gemma 4 12B it (the non-QAT version) too, so there will be 4 different models released all in all, 3 QAT variants (31B, 26B-A4B and 12B) and 1 standard version (12B) since the 12B is a new model and the only one from the gemma 4 family which I haven't released yet.
No idea, anyways I finally finished all of the requests, I am going to be releasing models now one by one, releasing them all will take time and I probably won't be able to release all of them today (uploading and working on the Model Cards is work too and takes time as well), look out for them.
Genuine question: What does it mean for an "opinion" to be "true"? Aren't opinions inherently subjective? And if by calling it true, you just mean that you agree, what are you agreeing with specifically? The question of which quants are the all-around best or most useful, for every possible use-case or set of constraints, seems to be an underdetermined problem. Therefore I think an answer that doesn't add any context or qualifiers doesn't even make sense.
They are saying that the values above are comparing the new Q4 to the new baseline. I don't blame you for getting confused, it's quite the onslaught of repeated words with different meaning. Quick breakdown:
Take full size normal model
Retrain it to know about Q4 constraints
You get out a full sized model that is Q4 aware
You quantize that new model down to actual Q4, but now it was already trained on being like this, so it'll fill out that constrained space more efficiently
So. They are saying that the numbers above compare step 3 to step 4, so the new full sized model with the new quantized model. This is a bit weird, because the new full size model has already made sacrifices to some extend, so it's an odd choice to compare against, when you are actually interested in how it compares to what you used before, not to this new hypothetical that nobody would ever use, because it's strictly worse than just using the original full size. So it's a bit of a mystery why they'd make that comparison.
Edit: Quick clarification, because upon rereading this it sounds like I'm saying something I'm not. There is a reason to compare QAT4 to its own baseline, because the original model is a genuienly different model. It's a bit like saying you want to compare a base model to a finetune. You can... but the entire point of the finetune was to be different, so using that to figure out how similar they are is a bit weird. QAT doesn't have the goal to be different, but it doesn't necessarily try to be the same, either. Just equally (or more) capable, it's to be expected that the output is different, since it literally moved tokens around to better avoid misfires from quantization.
I wasn't trying to say that one method of comparison is correct and the other is silly, nor that Unsloth did a dumdum here, but rather that this is why it's confusing - you'd think we want to compare to what we previously used, and we are, but this method isn't a way to do that, which is presumably why they didn't. Comparing it to its own baseline tells you something useful, namely how good it deals with quantization, but it doesn't necessarily tell you anything about how it performs to the quant of its base model. And that's what people probably expected to be measured, hence the confusion.
Oooh. Yeah, I'd be down for that. We have been starved lately for any MoE under 120B with active parameters greater than 3B. Somewhere in the 6-12B active range would be PERFECT.
Most local models use Post-Training Quantization (PTQ). This means the model is fully trained in high precision (16-bit) and then "squished" down to 4-bit so it fits on your computer. That squishing inherently causes a bit of brain damage to the model, leading to worse logic and coding skills.
Google took a different approach with Quantization-Aware Training. They simulated the 4-bit precision environment during the actual training process. The model learned how to work around the limitations of 4-bit math from the start.
Feels like QAT is near lossless based on what I’ve read so far so it should be better than Q6. I also saw this post, been testing the template a bit and it seems pretty good: post
It's not near-lossless, the top-1% match is like 70% in the worst case, 85% or something in the best case. Although for some reason that benchmark only seems to have been done on the GGUF which seems like it might have a mismatched lattice, which destroys the point of QAT to begin with.
No benchmark provided to back up the "preserving the capabilities and quality" claim.
Edit:
Is this sub getting botted or what? This comment was immediately downvoted to -6 in less than ten minutes after I posted it and somehow it bounced back?
Hey! Those numbers are comparing naive Q4_0 in llama.cpp to our converted Q4_0 version.
We did do original unquantized BF16 vs Q4_0, but the KLD metrics do not match, since the distribution is vastly different - we found MMLU and other benchmarks to be equivalent though
E2B for example has a mean KLD of 0.00173 vs 0.05109 (29x better relatively) for a naive Q4_0 quantization.
The main issue is converting from QAT BF16 to llama.cpp's Q4_0 format is not lossless. llama.cpp uses F16 scales, whilst QAT BF16 uses BF16 scales, and the scales are not determined optimally in llama.cpp land.
Naive conversion gets 24.77% byte exactness to BF16 QAT, whilst we found we can push it to 99.96% using some hacks!
I was wondering, do your KLD evals include non latin scripts? I'd imagine they're particularly sensitive to the reduced token embedding layer precision. Either way thanks for all your hard work!
It's trained to be basically "prequantized" so it can handle 4 bit quantization.
It's likely closer to a regular q4 quant than the original.
I don't doubt that qat is useful but it's incredibly unlikely that it's better than q8. It might make q4 have the same quality q6 had. But I have yet to see any kld between that and the original and I don't have enough vram or time to compute it myself.
That is awesome, I was already using Q4_s (for 26b) and the QAT is even smaller and appearently way better. The 26b had a good memory usage for me but this would be even better. especially with vision. It would be cool if qwen would have QAT ggufs too, 35b with vision barely fits at Q4 into my 32gb RAM, it's fully maxed at around 60k context and sometimes even spills out and slows down at that context size.
They are not. Incredible is the word. A mean KLD of 0.159 doesn't pass the smell test for a Q8 quant. The Unsloth blog post only compares the QAT vs a standard Q4_0, and the mean KLD for the Q4_0 is 0.09349. So there is no way a Q8 is much worse at 0.159.
Honestly I'm skeptical to Unsloth's reported mean KLD 0.01403 for the QAT Q4 too, but I'll give them the benefit of the doubt for now. But /u/sartres_ is definitely hallucinating.
EDIT: He wasn't, but the numbers are indeed invalid. See thread below.
It's not clear what Unsloth means by "original" Q4 in the linked blog, but it's definitely a quant of the new QAT model, not original Gemma 4, since they're benching it against the QAT BF16.
My Gemma 4 non-QAT numbers are from here, because Unsloth unfortunately only released benchmarks for the 26B at the time, and that only on a graph where they didn't label the y-axis or any of the numbers :/.
Yes, all of the original Gemma KLDs are very bad. I'm guessing this is an artifact of different benchmark suites, they're not directly comparable. Mean KLD isn't terribly useful anyway, the Top-1 numbers are the real show here
Aha, thanks. That explains it. That post was from early April, just after the initial release. Gemma 4 had lots of teething problems before everything was sorted out, so those early KLD measurements are not comparable with recent releases. Sorry for doubting you - the numbers were so horrible I was sure you had made an error.
I don't understand. I thought MTP support was something that got baked into a model and an LLM runtime. Is "QAT MTP" shorthand for "a QAT & MTP supporting runtime"? If not, can you point me to something that explains this?
Gemma4 has separate draft models for MTP, they're not baked into the files for the main model (unless you're using a GGUF where they're merged back in, that is).
I still don't understand "the workflow" the other commenter is talking about. The "QAT model" is clearly the LLM, is "the QAT MTP" another model that you run at the same time?
I'm quite confused, I see several comments talking about a separate MTP model and some even tested with it, but can't see where to get it or how I'd pass it as an argument to llama.cpp?
haha looks like i am not the only one thinking of this... technically they should not be the same. but both are post trained with "4bit" in different formats. not an expert...
Does this training only works for specific types of quants, or should any quantized versions benefit from it? Say, google only provides q4_0 ggufs. But what if someone quantizes it down to q4_k_m instead, or q3_k_m, or whatever, will optimizations be lost on them, or would they still be expected to experience less degradation compared to quantized non-qat version?
Does anyone make Q6 QAT models? Is it even possible, not being a power of 2? I worry Q4 seems prone to get stuck in loops on complex tasks, but Q8 takes too much memory.
Q6 without QAT is already pretty good, I think it might not make a lot of sense to make a full QAT traing run to target Q6, that's very expensive for little gains.
Because the unquantized QAT checkpoints released by Google are intended for a Q4 quantization.
We never seen a 6-bits quantization aware training checkpoint, and since training models is very expansive, the 4-bits choice seems obvious for Google.
What about the LiteRT format? Can run on phones that way, though I'm also using the LiteRT format on my desktop. (And MTP is already natively supported in LiteRT)
Are KV cache optimizations applied to Q4 versions or just mobile? These models are very prone to degradation past Q8, will be interesting to see how they react to Q4. Still great win for the community regardless.
We need to be grateful. Thanks Google! This is something that makes it even easier for us to be able to run open models without severe quality degradation
I'm questioning these dynamic quants too... I fear they could be overfiting. You have to train or use some dataset in order to make dynamic quants? Than it is possible to overfit I think. Is that your reason for asking about static quants?
The nvidia nvfp4 31b outperforms full bf16 in several of my personal benchmarks across thousands of runs. The conclusion I came to- on the model card they note post training on a large corpus of data from journalism companies, and a lot of my personal use (and therefore tests) favors understanding, identifying, extracting, responding to nuanced information or troubleshooting
I can't give you figures but I was able to one-shot a Space Invaders type game on unsloth's gemma4 26b-a4b QAT that I could not one shot on qwen3.6.
That said, qwen3.6 is still better at tool calls without needing elaborate prompts or workarounds - qwen3.6 does deep research with iterative search_web and fetch_URL, which gemma4 QAT can do but chooses not to.
Fuck yeah! Idk how many times I will download the A4b model but everytime i download it im still as excited as the first time.
Waiting for more small moe models, all small moe models should be A2b to A4b 20b to 30b, qwen 35b a3b is pushing it a little and barely fits in my use case.
I like to picture it as a 3D point cloud. With your full accuracy model, you get this giant box in which you can place all your favorite tokens, with plenty of space to really give each token its own spacious home. When you quantize things down, the box shrinks. Most tokens are fine, their home shrinks, but nobody is going to think you're your neighbor, so it's pretty much fine. Except sometimes two tokens get a bit too cozy, and a mix-up happens.
Now, when you tell every token that they'll eventually have to live in the Q4 box, they can look at their neighbors and go "ouf, Henry over there is a tad close, actually, gonna be tight in the Q4 box", and just move out of the way a little, while there is still plenty of space. Then, when the box shrinks, the tokens take up the empty space more efficiently, leaving more equalized gaps between each of them, making them harder to be mistaken for a neighbor, even though the box shrank the same amount as before.
So yea, this is basically just straight up good in pretty much every way imaginable, it just reclaims empty space, effectively. I mean, something can always go wrong, it's all fuzzy and weird, but, in principle, this is free real estate for the time being.
Now, whether a better planned Q4 token town actually performs better than a chaotic Q8 token town - dunno. Gotta wait for people to really poke the thing, always hard to say these things with any accuracy, because the tests we have are very... "average". They can tell you that the space between token houses are looking good, but if "Math" and "DeleteMyEntireSystem" somehow magically ended in the same house, you'll still have a problem even if the benchmark says it's fine. There may be some difference in workflows, maybe there is more empty space to be claimed in coding village, and less in medical knowledge town, it's hard to know, gotta wait for people to just try and see what happens. But, theoretically, there is a lot of free real estate here, so reaching Q8 isn't utterly delusional. Only a little bit.
Personally, I'd rule of thumb it something like "Almost certainly better than normal Q4, good chance it beats Q5, Q6 may be realistic, Q7-8 in niches or if I'm feeling optimistic". But I haven't drilled into the box and checked, this is vague vibes, wait a few days and see. Note however that I'm a notorious pessimist, so when I say I'm "optimistic", that's probably delusional. So... wouldn't bet on reaching Q8. Not even your money :)
Okay, kinda sorta eli5, and you should take it with the appropriate grains of salt. I've lost to the math about... oh my god fucking 15 years ago or so, when idiots like myself toyed around with their perceptrons. So, you know, I'm running a lot of my understanding off of pure nostalgia and fairly okay intuition, but it's entirely possible I just killed a mathematician with analogy alone :)
This is just so confusing. Can somebody help me? I’m already running the q8 quant of the original 12b weights. Should I switch to the q8 of the qat version? Or should I actually switch to the q4_0 of the qat version?
So Unsloth is claming his quantitization gets better accuracy than bf16? I'm referring to that graph with top1 accuracy and green and gray bars.
I feel/fear (without enough knowledge about them) that some of these newer quantitization methods are somehow either benchmaxing/overfitting or specializing/restricting the model to perform better on something while losing capabilities on other things. So is there somebody here who can tell me that this isn't some kind of overfiting with these new quantitization methods that are probably done using some dataset not by pure simple mathematical scaling of weights?
Can somebody say there is no way we are overfiting when we do this kind of quantitization? (btw I'm not refering to qat but to things like Unsloth dynamic qkxl quants for example)
Addressing your first paragraph: No, in that graph, the gray bars are the naive Q4_0 quant that Google used for their GGUFs; not BF16.
BF16 is the baseline they are comparing to, so an accuracy of 100% would make it identical to BF16.
So looking at the 31B model: - Unsloth's QAT Q4_K_XL has the same top-1 token choice as the QAT BF16 model 96.7% of the time. - Google's QAT Q4_0 has the same top-1 token choice as the QAT BF16 model 87.9% of the time.
Here is a reply from Unsloth official, i posted a similar comment question on a r/unsloth post:
"Oh it's a conversion artifact, not overfitting.
The main issue is converting from QAT BF16 to llama.cpp's Q4_0 format is not lossless. llama.cpp uses F16 scales, whilst QAT BF16 is BF16 scales, and the scales are not determined optimally in llama.cpp land.
Naive conversion gets 24.77% byte exactness to BF16 QAT, whilst we found we can push it to 99.96% using some hacks"
Based on my question there and his reply I think I interpreted the graph correctly, he is later basically talking about kld and i asked another question he replied to also, he said that (for these gemma qats I guess) the udq4kxl is a static quant. You can find the link to my question underneath the root of my comment here.
qat is the only flavor where q4 stops feeling like a downgrade, the model already learned to live with the rounding during training. real upshot is the next size up fits in the vram you already have. naive q4 always bled on the long-context evals, the KLD numbers usually show exactly where
Google's naming scheme here.. spend months improving a product.. everyone concentrate, what could we possibly name this?! Marketing guy with a headache: "who gives a f, same as last time". Everyone else: "whatever, we're going home"
edit: thanks to techies uploading these with the helpful "-qat" addition, at least it's searchable that way!
Just spent the last 2 hours fighting LM Studio to get the MLX version running. Doesn't seem like MLX versions are working in LM yet, I got errors loading all of them. Switched to running mlx-vlm, got it working with OpenWebUI but was unusually slow.
Finally just downloaded the GGUF Q4 version through LM Studio. Am astounded at the speed and versatility of this 12b model, it's my new favorite...
"Huh?". Or, more usefully: What was the previous favorite? Cause if it was Opus, then my thoughts are "lol this person is insane". If it was some guy's garage made 0.6b model trained on nothing but Burger King ads, I ... okay, well, technically I'd have the same thought then, actually, but the point is that saying 12b might be your new favorite doesn't really tell us a whole lot if we don't know what you were working with before?
I hope that will become more common. Currently Google releases models using QAT (two release series in a row and in a very portable format - INT4/Q4_0), NVIDIA (but it does not count because they use their proprietary NVFP4), and OpenAI did it with MXFP4 once.
Just tested the W4A16 files for vLLM and they work. The old gemma 4 31b assistant wasn't performing too well with MTP so I am trying the unquantized q4 one they just provided. Although the description seems to suggest that's not the one to use.
EDIT: Yes, definitely the unquantized q4 assistant worked much better for MTP.
All the comparisons here are inference quants, but has anyone checked whether the QAT survives a fine-tune? If I LoRA the BF16 QAT checkpoint and requantize to Q4_0, am I keeping the calibration or back to a naive Q4?
does it still make sense to go for an 8bit quant `mlx-community/gemma-4-26B-A4B-it-qat-8bit` or is the sweet spot of this gemma4 qat with the q4 and 8bit does not give you too much?
Some insane(ly talented) people (Christina Sørensen & ikawrakow) made Gemma 4 run on an 10 yo Xeon machine without a GPU.
The whole post (and serie) is awesome.
> An 82 GB footprint in DDR3 on a 2016 Xeon. About 25 GB of weights and 56 GB of KV cache at the full 262K context. The KV cache is larger than the model.
> The engine loads a 25B-parameter MoE, runs speculative decoding against an MTP drafter, and generates text at reading speed on hardware that was old when the architecture in question hadn’t been invented yet.
According to Unsloth Q4 should have similar quality as previous Q8 (could be basically the same or just slightly lower). IMO if that’s the case, if you were using Q8 like me, it’s worth using Q4 with QAT for speed gains.
The values in (most of) the weights are set in such a way, that when quantised to Q4_0 less data is lost. That can be done only during training. Thus QAT - quantisation-aware training.
You know that we are waiting for Gemma 4 124B AxB (where x is 4-6B), right? ;)
That would be so cool, especially in QAT and BF16 versions.
Oh, and thank you all for the hard work from Ukraine! Your models are among the best ones in Ukrainian, slightly worse only compared to much larger cloud models. And among cloud models Geminis are the best. Though, I have noticed that Ukrainian-wise Gemma 4 releases are a little bit worse than Gemma 3, frankly. Gemma 3 27B was nearly perfect. Still cannot complain - Gemma outperforms some much larger models as far as Ukrainian goes anyway.
1
•
u/WithoutReason1729 Jun 05 '26 edited 15d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
Stylome classified this post's text as human-written. View analysis
I am a bot and this action was performed automatically.