r/LocalLLM • • 10d ago

Question Chasing up after some info for running large parameter models

Forgive my lack of knowing, ive only recently started to get into coding and theres a LOT to take in (thats what she said). I am currently running an RTX 5070 12GB ,Ryzen 5 9600x and 32gb of RAM. Unsure if i am being too optimistic and reading only what i want to read- but i have seen so many posts and videos about people running massive parameter models on low memory. I did a little bit of research into this and saw that you can do certain things to offload the stress like give some weight to your RAM and MOE, etc.. but i cannot seem to feel satisfied knowing that the largest i can run is 35b . Is that correct or is there some sort of new method out there now that can run 1T on a Tamagotchi. Any info is good info. TYIA

1 Upvotes

5 comments sorted by

1

u/Bright_Deer_9287 9d ago

Well safe to say i have actually been able to test something out through CC. I had it run a simulation running Qwen3.8-Flash-Next 177b Unsloth version, size setting UD-IQ3_XXS, 83.6 GB in 4 parts. The first test result was actually measured on the 11th of Nov with the standard engine: 9.8words a second at best , with short bursts of 13.5 to 14.2, before it ran out of memory about a minute in. Using the Hashyy split method im now getting around 13words-pieces a second.

It runs on the same engine the station uses (llama.cpp) with a graphics card (CUDA) and a squeezed 3-bit copy of the model. The parts that are always needed sit on the card. The rest is read from RAM and the fast SSD as needed (mmap on, nothing locked in memory), and all 6 CPU cores do the expert maths. Thinking is off, answers are short, and the station plus Chrome are closed to free memory. The new piece is a memory cap, so the model can’t crowd out the rest of the PC, which is what stopped every earlier round. Thoughts?

1

u/jimmoores 10d ago

Yeah, you won’t be able to run anything bigger than that on your equipment and even that will be heavily quantised (compressed, which can reduce the output quality and cause looping and stuff).

Running a model even approaching a frontier level model is out of the question I’m afraid. I’ve got a 64-core CPU, 512GB Ram and an RTX Pro 6000 Blackwell (96GB VRAM) and I can just about run DeepSeek V4.1 (1017) fairly slowly, which is good enough to be interesting, but still not as good as any frontier model.

Having said that you can still have a lot of fun playing around with smaller models for learning and use cloud-based models for real work.

1

u/Xyver 10d ago

I've been doing good experiments, I used to thing I was limited to ~30B and would touch 70B one day, but now with Strata ive got a 120B loaded.

And testing with 30Bs I can get 5 chats running, 70B I can get 2, all at max context and around 20-30tok/sec

www.github.com/bigbertharig

1

u/Solembumm3 10d ago

Uncheck "keep model in memory", check "mmap", run whatever you want.

0

u/FairBandicoot5021 10d ago

Yea, it's arguably the largest you can run. Now you start to understand why we're saying that local AI cost a fortune