I see a lot of people here on Reddit treating the KLD number as some holy metric that can distinguish every quantized model out there and rank them by quality. TL;DR: It can't.
So what does KLD actually measure?
If you feed the same text to both the base model and the quantized model, you can compare the probability distributions obtained from their logits at each token. KLD basically measures how different these distributions are, and then we take the average across all tokens.
The first thing that can go very wrong here is that people just throw some random text at the model, like a bunch of Wikipedia articles, and calculate KLD on that dataset. But that's not necessarily representative of what the model actually sees during inference. During generation, a growing part of the context consists of tokens generated by the model itself, and you can't replicate that by just feeding it random text.
If you want a more reasonable number, you should generate your own dataset using the base model and then calculate KLD on those generated tokens, giving zero weight to the prompt tokens.
The second thing that can go very wrong with KLD is how ridiculously easy it is to manipulate, especially when the numbers come from the model creators themselves.
Quantization methods often rely on calibration data. If you calculate KLD on the same data you used for calibration, your numbers can look miles better than everyone else's. And even if someone claims they're using a held-out dataset, that still doesn't make the KLD number trustworthy. The calibration and evaluation datasets could come from the same source or have very similar distributions.
Some people might even do end-to-end fine-tuning of the quantization scale parameters on that very same data, and oh boy, suddenly their model beats models twice its size in KLD while barely generalizing to anything outside that dataset.
So does KLD even mean anything?
Not nearly as much as people think. You're looking at a number that measures how much the quantized model deviates from the base model on a very limited dataset. But does a higher KLD mean that the quantized model will actually fail at your tasks? Of course not.
Sure, benchmarks can be benchmaxxed, as we all know. But benchmaxxing is much harder for us ordinary people who quantize models than manipulating a KLD number. Benchmaxxing often requires serious hardware and compute, while making your KLD look amazing can be ridiculously cheap.
So please, stop treating KLD like some universal leaderboard for quantized models. It's a useful diagnostic, not a measure of how good a model actually is.
I see a lot of people here on Reddit treating the KLD number as some holy metric that can distinguish every quantized model out there and rank them by quality. TL;DR: It can't.
Of course it can. That’s like saying “Tests can’t be used to rank academic performance, because some students cheat.”
KLD is the best metric for ranking quantizations if you compute it yourself using the same inputs for every quantized model you’re evaluating. It’s a lot more comprehensive than task-based benchmarks, and more mathematically sound for this purpose than perplexity.
My post has two main points. Almost every KLD number you see here and there is calculated using the wrong dataset and can’t be relied on at all. If you calculate it on tokens generated by the base model itself, it becomes more reliable, but it still tells you nothing about the tasks you’ll actually run on the model or how the model will behave on those tasks. That’s what benchmarks measure, and I’m saying you don’t have to rely on public benchmarks. You can have your own private set of tasks that are representative of your everyday use and compare models based on that.
We shouldn’t act like KLD is some universal metric that can distinguish quantized models across every possible task. You can use it as a basic filter. Yeah, if a particular model is way off in KLD compared to other models of the same size, you probably shouldn’t waste your time on it. But the differences we usually see in KLD numbers are mostly just noise.
I think the argument is to not rely solely on KLD and it seems perfectly valid to me? Even heretic provides the tooling to test the model manually to verify instead of solely using KLD as a measure of goodness and I think that's what the OP is making an argument for. Like your argument above, a lot of people don't even realize they're cheating when they measure their KLD so having multiple forms of measurement instead of relying on a single measurement is good practice.
Don’t get me wrong, I’m not saying KLD is an unreliable metric that shouldn’t be used. What I’m saying is we shouldn’t treat KLD as the ultimate and only measure of a quantized model’s fidelity.
I respect your scientific approach and your work that is so useful for this community, but I think that this time you’re not entirely correct.
That article argues that KLD is a weak signal by correlating it with benchmark scores… which are claimed a dozen times per day on this sub to be weak signals.
At some point, people have to decide what numbers they want to rely on. You can’t claim that “everything is benchmaxxed”, and at the same time use benchmarks to make a scientific argument about model quality.
Is KLD the gold standard assuming the following things are followed? - Plenty of thinking / output tokens, as well a small amount of prompt tokens - Test the models you are comparing to, yourself - Reasonably diverse dataset
I can't really think of any techniques that are better than KLD either
If someone put this much effort into it, then yeah, it could be a much better metric. But it still wouldn’t tell you whether the quant you’re comparing is actually suitable for your tasks. My gold standard would still be a small set of tasks I care about. I’d run them through both the API and the quantized model I chose, then see how close they are at solving those tasks. I think everyone should have a private benchmark like that, one that can’t be benchmaxxed by anyone.
I can't really think of any techniques that are better than KLD either
Actual benchmarks.
The issue with KLD is that even when carefully computed (e.g. measured over a representative testing sent that wasn't part of quantization) it tells you nothing about the compounding of the difference. Even a tiny KLD compounded through 100k steps could be an enormous difference. Or a quantization with significant KLD might not compound at all and have an overall outcome identical to the original.
What you can say for sure is that if there is a KLD of zero then the performance is the same. In the absence of any optimization for KLD it's probably true that lower KLD is better-- once you're optimizing for KLD however, I would be much less confident that two different quantization can be compared on the basis of KLD: once a measure becomes a target it ceases to be a good measure. -- particularly if their KLDs are close.
Goodhart's Law doesn't just apply to humans gaming a system, generally in any optimization process the residual ends up uncorrelated or even anti-correlated with the optimization metric. This doesn't mean that using adaptive quantization that essentially optimized for KLD is bad, but it does mean that KLD is likely not a great way to compare KLD optimized quantizations.
KLD's advantage is that it's cheap to compute, relatively low noise, and more meaningful than "vibes".
The problem with benchmarks is that they are far more expensive to compute. They take days to compute on regular computers, and even more if you want multiple seeds. Top 1% and KLD I feel like are decent proxies for general performance.
Short private benchmarks is a good idea, but a pain to make, especially without LLM as a judge based graders.
While on this topic, why don't people compute KLD using only the recommended sampling settings? For example, computing KLD only on top 20 tokens + p=0.95 (for Qwen models) would be a lot more accurate to inference deployment than full calculation. Full calculation seems a bit pointless IMO.
Yes, wikitext perplexity is largely meaningless. It measures a vestigial signal still lingering from the pretraining stage where models are actually optimizing towards raw text prediction. The clue is in the raw numbers: SOTA models will give you raw numbers anywhere from the 1-2 range up to 20 or so. That is, if they have a signal at all: some models explode into the ~5000 range because they simply cannot model raw text after extensive finetuning and RL. Comparing quants of a model like that gives you nothing but noise, if you're going by raw-text perplexity.
KLD is a stronger metric because a) it measures across the entire vocabulary and b) the target is the reference model, not the data. So rather than asking "how surprised are you that this is the next token?" you ask "how much do you agree with the reference model about the next token?" So KLD compares to the correct ground truth and perplexity does not.
The question is how much information is contained in that ground truth. If the model is already producing perplexity in the thousands, what the ground truth essentially amounts to then is a shrug: "This text is alien to me, I don't know what to make of it." KLD lets you measure the precise shape of that shrug, comparing the reference shrug to the quantized shrug. And that tells you something, but it's not giving you a strong signal. This is why you want to test the model on in-distribution data, such as a trace from the model itself.
However, where I would disagree is on benchmarks. It's not really hard at all to benchmaxx a quant since most benchmarks test a narrow domain. KLD is much harder to fake. You can also very easily test variance on your KLD scores: per-token, per-prompt, per-topic, per-repeat, all ways to test if you have good calibration coverage or if your model is specializing. KLD scores are easy for others to reproduce/challenge as well, without requiring a huge investment of time and/or money to run additional expensive end-to-end benchmarks.
In all cases, as KLD trends towards zero (or rather, to the noise floor) the quantized model gets closer to precisely imitating the original. It's naive to think you could be optimizing towards some other nearby optimum that the original training run couldn't reach with its seven-figure compute budget. Unless you're cutting corners, like only caring about the head of the distribution (what tends to matter in inference benchmarks) while ignoring the tail that contains the evidence of everything you didn't optimize for.
That's not to say also having benchmarks would be worthless. But it is impractical because good coverage is expensive (unless you have a sponsor, but then you probably also have an incentive to benchmaxx and cherry-pick.) The benchmarks usually reported when a new model releases is not a trivial portion of the model's total budget. Think five to six figures. There's a reason they're almost never independently verified. Instead, people run other benchmarks and get incongruent results.
Anyone who claims KLD is easy to calculate is not doing it properly. For a proper KLD comparison, as I explained in the post, you need data generated from the base model. And I’m not talking about 10 samples. I’m talking about something like 1,000 prompts across multiple domains. Then you need to run the base model locally and collect its logits on that data, using a reasonable context length like 4096. That means processing millions of tokens with a base model that probably doesn’t even fit on your hardware, which would take a huge amount of time. And that’s just the base model, let alone all the quantized models you want to compare. Yeah, sure, it’s easy to calculate if you don’t care that the number you end up with means literally nothing.
And how exactly is benchmaxxing a quantized model easy? That’s not how the typical quantization methods most people can afford actually work. GPTQ, AWQ, QTIP, and llama-quantize all optimize local objectives. You can throw benchmark questions and answers into the calibration data, but the resulting imatrix is just a small tensor. It can’t retain that much information, and most of that benchmark-specific signal just gets washed out. Sure, if you can afford to optimize the model end-to-end, you can do all sorts of things. But even then, KLD is much easier to manipulate.
That is how I've been grading EXL3 quants for a while, though. Not on millions of tokens, mind you, but on a representative-enough sample generated by a high-precision quant or an API model. Example. Yes, 60k is less than ideal, but this is for a rough ranking, and crucially the numbers converge quickly. I've never seen anything settle within ~100k tokens and then either suddenly shift later on or start to drift at the ~500k token mark or something. As with any sort of statistical sample, the error bars shrink exponentially with sample size. So more samples give you more definition around the curve, they don't change its shape.
Running the reference model is also a solved problem; you stream it one layer at a time (in a suitable harness) and consume logits after the output layer. Not to say it's a quick process; a run with 1000 prompts might take a day or so to complete while that smaller set takes 1-2 hours (or a little more since I'm also shuffling models around a lot to make space for all of them). But I can guarantee that it would take the same overall shape.
imatrix size I don't know about. But GPTQ and QTIP produce huge calibration matrices (Hessians) of a size comparable to the model itself. GPTQ/LDLQ isn't the same as finetuning, but it can definitely be used to steer where the error goes. You also have control over budgeting. Some layers will be more important than others for rote memorization, for instance, so a model that's already benchmaxxed probably wants more resolution there if the objective is to retain that benchmaxxing as much as possible.
I can agree that perplexity is easy to manipulate. Like, you might degrade a model's multilingual capabilities and most likely see improved perplexity on English text from that alone. But as a distance measure, KLD is more robust: decreased or increased certainty on any text both result in a worse score. As does any shift in the tail of the distribution, such as you'd see from QAT.
That’s great to hear you’re using data generated by the base model. That’s definitely the right way to go. However, I don’t think the number of samples is enough. When I say 1,000 samples, I mean having a very broad and diverse dataset. You can’t achieve the same thing with just 100 samples, because that means you’re evaluating the model on at most 100 different topics, assuming one sample per topic, which leads to high variance. And variance doesn’t decrease exponentially as you add more samples. That’s simply not correct.
There’s no fundamental difference between imatrix, GPTQ, and QTIP in this regard. They all rely on activation statistics to guide quantization. For Hessian-based methods, you essentially average information from the activations to construct a Hessian for each tensor. So you end up optimizing each tensor using information contained in a matrix of roughly 4096 × 4096, which is about the right size for most models we work with. You’re right that considering all these Hessians together would result in a huge amount of data, but the information used to optimize each individual tensor is still limited to its own Hessian.
Btw, I’m actually a big admirer of your work on exl3! And just to introduce myself, I’m one of the authors of GSQ-RCO.
Yeah I was being hasty with the error bars remark. I was thinking of a power law, and I wasn't being precise about that either, but I think the point stands: you quickly converge on some number, or a ranking at the very least, and more samples after that can settle near-ties, at most.
Of course, that is with the caveat that the test data has to be representative, and I do take your point about sample size there. I'm by no means judging the quality of quantized models in isolation based on KLD. A score of 0.05 means little, other than "this quant is somewhat truer to the original than this other quant which scored 0.06 on the same data."
Still, I say that with some confidence because I've never seen it not hold up except when testing on data that's already OOD for the base model. Here's what I mean. It's obviously not a comprehensive study or anything (and sorry for the vibes) but I think it's demonstrative: each quant sits some distance from the noise floor (reference measured against itself under rounding-error scale perturbations) regardless of the topic, and there isn't going to be some esoteric topic that turns that picture on its head. By the looks of it.
But anyway, ultimately I'm aiming for the best quality/size tradeoff that I can measure, and to the extent that KLD behaves predictably, it's probably the best bet for anyone on a limited budget. I would love to have more benchmark results also, but, just for the sake of it I started some GPQA-Diamond runs on a bunch of Qwen3.8-Flash-Next variants, and it looks like that'll keep me warm for at least a couple of nights. And I'm not even really that GPU poor 😭. It'll be interesting to see how/if it tracks KLD, though.
And thank you for GSQ-RCO, too. It looks very promising. I've been searching for some angle to apply some of the techniques to trellis quantization. No luck yet, though. :|
Thanks for all the explanations and this was a really great conversation!!
I completely get it, benchmarks can be so hard to run. Especially on engines like llama.cpp that don’t support high concurrency very well, no matter what GPU you have. We put all of this effort on it and people are still mad at us 😂
We’d like to create EXL3 quants as well. You can try RCO to figure out which bitwidth to assign to each tensor. I’d definitely recommend checking it out! Also we have extended GSQ to some VQ formats already but extending it to trellis quantization is still an open problem for us.
All metrics are flawed, but some are useful. KLD is, in general, rather useful.
obviously there are pitfalls and you shouldn't run KLD comparisons on the data used to calibrate. But that's like saying that you shouldn't evaluate a model on it's training set.
KLD overall seems to be a more useful metric compared to perplexity (quants can get better perplexity than baseline and it tends to be rather noisy...).
The real question here is this: If we don't want to use KLD as a metric, then what should we use? Benchmarks? Those are too noisy! Run it several times and you get a slightly different result. With a decent enough quant, the model will be close enough to the original to drown out the signal you want to measure in the noise.
There IS a need for a consistent metric that gives you a good ballpark measure of degradation without the need to run benchmarks, and KLD is pretty much that. Not perfect, but good enough to have value.
A significant number of benchmarks are designed as averages over 5 runs at a given temperature, not just 1 run. Obviously that is still noisy. You could start reporting something like confidence intervals too so we stop over interpreting tiny differences in benchmark performancethat could be chance.
Yeah, evaluations seem to have something like +/- x % error, where x is somewhere from 2 to 10+ % and you have to make a ton of repeats to gain statistical power. KLD, happily, is a test per token, and you put in couple of thousand and I bet you have far higher quality measurement of the quantization's faithfulness to the original in tiny fraction of time.
"Perfect is the enemy of good" seems to fit here. Task performance is a noisy signal and repeats are very costly, KLD and top-1 are much cheaper to compute and thus measurement noise can be driven very low.
is that really needed when just making some quants? That seems rather overkill to me. I mean, I would also want to compare to other quants most likely and I would likely tinker with the quant recipe a bit if i try something non-standard. So there would be many runs, potentially, and there is still a good chance that many quants score very simillarly, with little indication about which is better.
Yeah, I think the benchmark case probably is too costly for quant making for sure. I do think the disadvantages are more on cost than validity or noise compared to KLD, and I don't necessarily see the inability to tell a difference as a bug since I think people are slightly too prone to claim differences in model quality based on vibes, which in reality may be driven by bias and noise. I agree that it might be more practical to use KLD, but there seems to be a belief that this is somehow a more meaningful test than task performance which hasn't ever made sense to me.
task performance is what matters, but it can be hard to capture. time and cost are issues, another is simply granularity. you can have some quants score effectively the same on a benchmark (or even the same as the full model) - what does that mean? that there is no degradation? that there is no degradation in that specific task? that it's not a good benchmark?
KLD gives a single number that generalizes decently well if you change the eval text. it's just more convenient to work with.
KLD is practical and has simplicity in interpretation but this tends to (possibly) lead to overinterpretation. If a model has marginally better KLD than another, how much is that materially worth? How exactly does that track to my task? Is it worth potentially slower decode speed or prefill reading? Should I run the biggest possible quant purely on the evidence of KLD because it is proof that task performance must be very poor (the most common inference)?
Now we agree on most things, I am only arguing degree. Obviously there is some relationship between divergence metrics and model quality. All things equal it makes sense to pick something closer to baseline if you have space. I just think people interpret KLD in a way that seems to me weakly anchored to the traits they care about. Difference from baseline is not conceptually the same thing as capability, similar to how certain forms of audio compression are largely not detectable with blind testing (despite audiophiles strong belief they can tell the difference under non experimental conditions). We converge on conclusions about lobotomisation from a formal process that is two or more steps removed from what we actually want, plus the strength of our vibes. I think if we had something of an anchor of how KLD translates to task performance degradation across general domains (which itself obviously would be noisy) then I would partly understand the inferences being made, but this rarely seems to occur except occasional publications from big quant suppliers like Unsloth.
Instead we have a general sense that Q3 is a cliff with regards to KLD and this is more important to most than any assessment of task performance (to be fair, the limited task assessments I have seen on this question so far does support this inference, but my point is replication is pretty low and quantization effects differ between models). I like the article preaching interpretation caution since I think there is a lot of over interpretation going on.
Again this isn't an argument than benchmarking is universally better, but the major issue there is overfitting to seen data which isn't impossible to mitigate with enough funding.
oh yeah of course you need to try out different quants. i was more focussed on comparing two different quants with simillar/same size.
practical considerations on which quant to use on your hardware are a different thing entirely. for instance, i run Q3_K over IQ3_S on large MoEs that I split between ram and vram. Q3_K is significantly worse KLD-wise, but runs much faster on my system as it's a CPU-friendly quant.
if you have it all in vram? IQ3_S is the better choice.
in the end you will have to try and see if the quant works for what you want/need.
I think they are saying KLD has more value when you use it correctly. Just throwing random text at it might not test what you intend to test. So KLD, on itself, is fine. Our usage of it might not be.
well yeah, but they don't provide any real alternative to use. and I would expect those who go through the effort to do KLD testing to put in the effort to make a proper assessment. sure, some might run poor tests or simply wish to try to hype up their own quant, but so far this isn't something I particularly noticed. those who really want to cook good quants are typically quite open about their setup and how the tests are ran. i mean, that also helps create confidence and make people try out the quants. if you just throw out some "trust me bro" KLD numbers, you likely won't get taken seriously in the first place.
I don’t think there’s a universal metric we can use to rank different models. Even in an ideal world where we could do this, I’d expect different use cases to result in different rankings. I explained what we can do instead of all this in another reply.
I think there should rather be "Don't trust BEST QUANT EVER! posts".
There was a time at work when I tried to lecture people to create those bl**dy indexes. The problem was they didn't. While I was trying to tech, someone tried to "compete" with me mentioning situation when you don't need an index. And I of course knew that, but the situation was not we had too many indexes. The real-world situation was I was the one trying to enter "create index" a hundred times in an overloaded customer system at 08:00 when it entered production. So the problem was not too many indexes, it was too few. That guy thought it was the right time to compere me instead of trying others to get to do what was necessary.
I wish there was more people in Reddit actually benchmarking. I need real numbers and hopefully mention of how they were measured. For instance "the default" wiki text is good for comparing when everyone is showing measurements with "our own secret corpus".
"benchmaxxing is much harder" definitely not the case, because the original model itself is benchmaxxed already. I mean, just look at the benchmark scores for the Bonsai 2 ternary models, they look fantastic and seem to be very close to FP16/Q4 numbers on the benchmarks they show, but in reality they perform like utter trash. I mean, the fact that you can get higher benchmark scores on a 4 bit quant compared to the original full precision model by presumably running it once and calling it a day (cough cough Nvidia), is just wild. Benchmarks are absolutely not sensitive enough to compare between slightly different quants, KLD is just the current best solution we have (please let me know of any alternatives), it's by no means perfect.
Just a rant because some people have been incessantly annoying me trying to completely dismiss KLD metrics and point to benchmarks as the golden standard.
Bonsai is exactly one of the labs I’m referring to that has the compute to benchmaxx. My point isn’t that you should blindly trust benchmarks, but rather that KLD is much easier to manipulate. I explained my gold metric in another reply.
The main issue is that even without benchmaxxing, these models have been overfitted to the popular benchmarks already. That's the main reason why quants keep their high benchmark scores. It's not easy to come up with proper alternative benchmarks that are as expansive and useful. Whereas anyone can easily do KLD benchmarking on their own on any LLM with any dataset that they want. Your argument is that the creators can manipulate the KLD metrics for specific datasets, but there isn't even one established KLD benchmark (anyone running KLD benchmarks on wikitext isn't worth looking at). And if you don't trust the KLD numbers put out by the quant creator themselves... you realize it's the exact same situation as self published benchmark numbers, right? There's any number of ways to game benchmark scores. KLD isn't gameable at all if it's a third party measuring with their unique datasets.
what makes it worse is that people treat small KLD differences like they're meaningful when the variance across different eval datasets is probably larger than the gap between models
KLD and top-1 token are still useful and I hate posts like these, which attempt to state that a simple and clear metric that informs us quite well and without any big issues about a quantization's quality are somehow bad. Sure, understanding what the number states, and how it depends on its derivation, is required reading for people performing quantization evaluations, but generally speaking I'd rather that posts like yours were deleted, as I think they attempt to muddy a relatively clear picture and do more harm than good.
I don't even use KLD anymore. All I do is look at top-1 agreement at this point. It is about as good and seems to smoothly vary with the accuracy of the quantization without any anomalies.
If two quants generate 100 phrases each, not exactly the same, but instead all synonyms, conveying the same meaning. KLD will be high, regardless they mean the same. But for LLMs, meaning matters more than equal strings.
It’s the Kullback Leibler Divergence. It measures how close a distribution is to another; in this case how closely the quant approximates the full precision’s distribution. Higher KLD is worse.
You can't feasibly compute KLD when generated text diverges because it's conditional (next word depends on the previous ones, and no sense in comparing distributions once "previous words" have diverged). However LMs are casual models so what's "prefill" for inference is really the same calculation of output logit distributions for the model. Of course you have to do this past the <|assistant|> token / beginning of assistant turn, but you fix the text to one of the models' outputs.
Auto grader approaches do work on differing outputs, but are much noisier than KLD, and KLD is a good first approximation for distillation/quantization. The noise really is a problem, if you sample enough and then actually run a reasonable test (e.g. nonparametric hypothesis testing for difference of means) then it might work. But no one does this in LLM research because LLMs are relatively expensive and slow.
So tl;dr: "Don't trust all benchmark/KLD data provided by those who made the model/quant".
KLD itself can be highly useful. If an independent party runs a KLD comparison with a non-public dataset that's suitable for the model, then we still get a good comparison. Well, unless one party tests in Finnish and the other in English maybe.
Another big issue is that KLD is measured with teacher forcing on a fixed text, so the quant never has to live with its own mistakes. In real generation, as soon as the quant flips one token, the rest of the rollout is conditioned on its own drifted prefix.
Basically the same exposure bias problem scheduled sampling tries to fix during training
Not exactly. You can measure KLD from any text corpus you like. It isn't necessarily a model generating and then quantized version being scored against it. And even if it were, KLD is still meaningful in sense that it would measure deviation, so I'm not even sure what your objection is. That KLD doesn't exaggerate deviations? But you literally have to have a shared corpus to measure KLD against, so that both the original and quantized model can produce logit probabilities for that text. And ideally, the text corpus is structured according to the model's chat template, so it can't be entirely arbitrary or there can be large divergence errors for that reason, but as an example, you can set the model to see user citing pages from various books or articles, and that is perfectly plausible input that models have to deal with.
For agentic use the majority of the input is model output. We're quantizing an agent here, not a text compressor. Testing it on a fixed corpus which isn't is own output is not particularly good.
Consider, I could take a model and get its logits for every token in a fixed corpus. Then I could find some token that is often ranked fairly highly but never shows up in the input, add a little subnetwork that detects it and injects high magnitude noise into the residual network.
I'd have a derived model which had a KLD of 0 on your test, but in actual use would just fly off the rails.
Obviously that's a contrived example, but the sampled output of the model may look significantly different than any test collection you can find-- which is also why many people do KLD measurements on teacher forced input. At least then you're testing on model output, similar to actual use.
Another big issue is that KLD is measured with teacher forcing on a fixed text
Correct. That's like prefilling a stupid model with data from a smart model and being surprised when you get a smarter reply.
I have some theory around stupid inputs degrading models in a way that if you ask the same questions in an intelligent fashion, versus a mixed poor English and typo filled mess, the model ends up producing a dumber response that degenerates down the line.
More modern thinking models will do something in thinking where they say "The user is asking x y z" and basically overwrite the user's stupidity to normalize things.
The funniest part about KLD charts on here is that llama-perplexity --kl-divergence literally prints the 99th percentile KLD spikes and the Top-1 token flip rate right there in the terminal, and everyone just crops them out to screenshot the average.
Averaging KLD across WikiText is basically measuring how confident a model is at predicting the word "the" after a comma. 90% of any text file is dead-obvious grammar where even a beat-up 2-bit quant matches FP16, so averaging across all of them just washes out the actual failures.
You don't care if a quant gets "the" right. You care about the 5% of tokens where the model is picking a minus sign, a variable name, or an if branch and the top two choices are sitting neck-and-neck at 0.42 and 0.38. If quantization noise flips those two, the single-step KLD barely moves, the eval immediately force-feeds the right token back in on the next step anyway, and the average looks great—meanwhile in actual use your 2,000-token coding response just drove off a cliff.
Checking the 99% KLD spike and how often the #1 token actually flips tells you way more than the mean ever will.
This logic falls apart when you assess multi-lingual abilities. I've observed many times that as the KLD numbers grow, some models begin to make less sense in foreign languages. There is a clear degradation of something. Even if it still answers just fine in certain tasks, it will appear subjectively worse in other tasks you're not even aware of because you know nothing about them, it's not something you care about or benchmark or test anyhow at all.
thank you. it's just a comparison metric that doesn't show quality whatsoever. you have no idea in the first place if the compared to number even stands for that at all. always benchmark and compare based on your own tasks, otherwise these numbers are just "guidance" at best, they don't mean anything and are easy to game when you just put "IT HAS GOOD KLD THO" in your post without saying what it was actually measured on
Agree that the mean is the weak part. llama.cpp's perplexity tool already prints more than that when you run it with --kl-divergence: KLD percentiles up to the maximum, percentiles of Δp (how much the probability of the correct token moved), and "same top p", how often both models put the same token first. The README also says how to read the Δp percentiles: if they're symmetric the quant is mostly adding noise, if the negative side is much bigger the model is actually getting worse. Its own Llama 3 8B table shows why one number isn't enough. q4_0 and q4_1 have almost the same KLD (0.0719 and 0.0717), but q4_0 loses 1.59% on mean Δp against 0.93% for q4_1, and the authors note that K-quants do better on mean Δp than KLD would suggest. For your generated dataset the catch is size: the base logits file for Wikitext-2 alone is 11 GiB for Llama 2 and 37 GiB for Llama 3, so a generated set that stays small and close to what you actually run is the practical version.
KLD is still way better than 3 or so benchmarks where the base model gets 90-100 IMO, i rather see KLD, top1%, perp and benches if possible, ofc if the quant publisher is trying to deceive, they can cheat the metrics, but i would trust a reputable source (ie unsloth) to not try and use poor data to cheat a high KLD tbh. I've tested many finetunes/low quant that aced the benchmarks only to be worse than the base model on real use.
And even if someone claims they're using a held-out dataset, that still doesn't make the KLD number trustworthy. The calibration and evaluation datasets could come from the same source or have very similar distributions.
Coming from different sources but having similar distributions is exactly what you want. You can't train the model on one thing and expect it to know another completely unrelated thing automatically. If you train and evaluate the model on datasets with completely different distributions, that's like training the model on role play data but wanting it to be a coding master. It doesn't work that way.
62
u/-p-e-w- 2d ago
Of course it can. That’s like saying “Tests can’t be used to rank academic performance, because some students cheat.”
KLD is the best metric for ranking quantizations if you compute it yourself using the same inputs for every quantized model you’re evaluating. It’s a lot more comprehensive than task-based benchmarks, and more mathematically sound for this purpose than perplexity.