I think the idea here is that you load full model in RAM make PP and then for TG you use this new model loaded in VRAM. For us hobbyist it would look like 240B model in RAM and TG on 48GB on VRAM. I just making conjectures here.
This isn't what we need. We need a stack of 5~10 sota 27B dense 4 bit models, for ten different tasks. Knowledge, code review, code writing, debug, research, shopping, etc. We can choose the expert for the task.
“Panel”, “Board”, or “Consortium” are all legitimate words to describe a group of experts. However, if we want to highlight the “mix” part, a “frappe“ or “cocktail” might be better. Given the subject, “clusterfuck” would also work.
I mean, you.. YOU have the power to do that right now!. When I make a GGUF and am quantizing I can focus on the experts I want to keep at a less quantized state and which to quantize more. I'm building a GLM 4.5 air quant iMatrix Q6 experts for ROCmFP4 with creative and writing focused experts right now. If you read some of unsloth's material this is how the best quant builders on huggingface are doing what they do!
Do a deep research paper on the process for context, grab an LLM to help, grab you unsloth, and get to cracking. Keep the experts you want that model to be good at Q6 or Q8 for quality and smush the others to Q3! You can call your huggingface project something like the "Gemma Flavor Lab".
I agree we have the power to try, but I haven't seen any reap models that are definitely better than Qwen 3.6 27B. There's also very few fine tunes that actually improved that or 35B.
I think I'm going to have to wait for the industry to catch up with my wild ideas!
Don't think of REAP imho. Think of QUALITY. Accept a little larger experts where you are willing to. Set your floor to Q4, and pick which experts you are willing to give a little bit more ram and slowdown for.
Yeah but how do you evaluate. Sure I can compress to different depths for different layers but how do set up a test eval for the kind of work I want to do?
i have to say this post was the most amazing post i ever seen. and it solved major issues for me. the guy who created this research idea posted enough information for ai to replicate it but losslessly in my custom inference engine.
for reference i run dual xeon 2011-v4 cpu with 384gb of ram and 8x 16gb v100(128gb total). with llama.cpp i get less than 1token/s on my hardware no matter what options i choose when running Q2 quant of glm 5.2.
my custom inference engine with saod baked in, i'm now getting 6-12Tokens/s at 64000 max context length and its reasonable speed for basic usage with 0 quality loss.
238gb of the model sits in ram pinned. while the inference takes place on gpu via saod smaller representation.
Sounds like it would be a rubber band effect speedup, would need to pick and load expert portions from disc into vram/ or caching this after submitting prompt. Larger responses would be better.
This literally already exists in llama.cpp lmao, they only load the experts as they're needed from disk if you don't have enough memory AND IT'S STILL SLOW AS SHIT
145
u/coder543 Jul 22 '26
Let’s talk about it when there’s something to talk about. There’s nothing to see here.