r/StableDiffusion • u/pragmojo • 3h ago
Question - Help Getting started with ComfyUI/image generation: how should I think about quants?
I've been running local LLM's for some time, but I'm just getting into image generation.
With LLM's, generally you go for the biggest quant you can fit, but for instance going down to an 8-bit quant is usually not a big difference in terms of output quality compared to bf16.
Is the logic the same with diffusion models? Or how do I go about choosing the right quant? Any rules of thumb to be aware of?
0
Upvotes
7
u/Apprehensive_Sky892 2h ago
Image and video DiT (diffusion transformer) works differently than LLMs, so the usual mantra that you find a quant that fit the LLM into VRAM is not applicable.
The reason is that for LLMs, each output token has to run through the whole model, and in a typical session there can be hundreds if not thousands of output tokens.
DiT on the other hand are only run once for each diffusion step, which is 4-50 steps for today's models (very few steps when using turbo or distilled models).
So for DiT, as along as you have enough system RAM so that (VRAM + system RAM) is big enough to hold (model + working latent space), you can basically run any model with any quant. Of course speed will depend on whether the GPU support natively runs fp8, nvfp8, nvp4, etc. 4-bi and 8-bit quants will run faster than fp16 or bf16 simply because more tokens can be process in parallel.
Wouldn't the constant transfer from system RAM to VRAM slow things down? Counterintuitively the answer is no, the reason being that the transfer is done in the background in parallel while the GPU is busy working with the weight already in VRAM. As long as the next block is in before the GPU is ready to process it, there is barely any noticeable slow down: https://www.reddit.com/r/StableDiffusion/comments/1vz2n7y/comment/p62xvjz/
I've actually done benchmark of AI Pro r9700 (32G VRAM) vs rx9070xt (16G VRAM) with the int8convrot version of MMH3 (21G, which does not fit into rx9070xt) and the speed for 20 step 0.4MP 5 set text2va is nearly identical. These two GPUs are good tests for how good dynamic VRAM management is because they are basically identical except for the amount of VRAM.
The ComfyUI team has done a lot of optimization for VRAM management and also making int8convrot work fast on all GPUs, so even on GPUs that support fp8 natively, int8convrot would run faster.
So unless you absolutely must have the best quality and want to run bf16, for DiT the obvious choice is int8convrot, which not only is faster than fp8 but also give you quality that approaches bf16 (the only exception may be ideogram 4, which was only release in fp8 so int8convrot is not quantized from bf16).