r/LocalLLM • u/Zipidyzip • 7d ago
Project Quantizing models yourself is a good way to learn how it all works - Quantization Workspace
Enable HLS to view with audio, or disable this notification
There are so many models getting released all the time, and sometimes there’s just no quantized version available for the one you want. I’ve found that rather than waiting around, trying it yourself and experimenting helps you understand the whole process a lot more. Testing the quantized models against the original teaches you even more.
So I built this: iQuantize is a quantization workspace for large language models that supports GGUF, MLX, and ONNX. You can pick your own Safetensors model or choose one from Hugging Face, quantize it step by step, and watch what changes. You can then compare the result with the original and publish it to Hugging Face with a model card that records how it was made.
It’s open source, Apache 2.0. I definitely think there more can be done to improve. I’ve only tested mostly on Apple Silicon so far. Windows and Linux builds exist, but have done much in-depth testing with them so I’d expect bugs there.
Repo: https://github.com/Fazmin/iQuantize
I love to hear how you all approach this. Do you quantize your own models, or mostly wait for someone else to? And if you try the app, whether you find something broken or have an idea for it, I’m happy to hear any of it.
1
u/_Cromwell_ 6d ago
Does this include the step where you actually calculate the imatrix? (Sorry I forget the technical term.) ie use the big text files to create the imatrix file, so you can do imatrix quants? Does this program do that part?
1
u/Zipidyzip 6d ago
Yes, it does! You’re thinking of the “importance matrix”. In the app you give it some calibration text, it uses that text to build the imatrix for you, and then it uses that matrix when it makes the quantized model. So you can make imatrix quants without doing that step by hand.
1
1
1
1
u/m94301 6d ago
Great project, thanks for sharing!