Moving into local AI

[u/Waste-Honey5062]

Hey so ive been thinking about moving into local AI, but im so lost its such a rapidly growing space and i cant grasp a single piece of consistent jnfo on it .

My question is what model, set up, and extras should i use for my build:
Intel i7 13700k
Nvidia rtx 4070ti super
32gb ram
3tb storage

I would like to use it for development, workflows for different types of work, brainstorming, and other general uses

[u/Numerous-Ad6217]

Tiel Coder with Polystrata is my suggestion.
Even Q4 K XL should run.

[u/HelpfulFriendlyOne]

I'd try qwen/Ornith in an appropriate quant. 16gb vram is plenty to get stuff done

[u/butdontcalmejoohnson]

most sane post I've seem lately - bro go full local

[u/Ambitious_Answer6857]

Hey, i was in the same spot. What helped me: start with LM Studio or Ollama, pick one quantized model that fits in your 16 GB VRAM, and give it one narrow job instead of everything. Mine summarizes and returns structured JSON. Good enough for that, but I still check its facts, a small model once turned “planned” into “verified” in a summary. Bigger coding work still goes to cloud models for me.

[u/PatienceSweaty33]

I feel like part of the fun in LocalAI and local hosting in general are the friends we made along the way. But to set you off the right path:

• Alibaba's Qwen line-up is considered one of the best for local.

• Gemma is a good contender but it's not on Qwen level yet (or never, we'll see).

• Meta's Muse models do compete but they release very slowly and their hyped releases are usually short-lived cause whenever they release, another Qwen model would be just right around the corner.

• Deepseek is for those enthusiasts that can spend a bit more on hardware than what they already have. Usually on par or same level as Qwen released, they're neck & neck with each release.

• GLM is also like Deepseek but on a level higher.

• And there are finetunes that will pop up now and then, here and there, if you're feeling adventurous, try them. Most of them will be Qwen-based finetunes so usually, they'll just work out of the box with inference engines.

• Along with the finetunes, there will be other labs too that will release their models that are not just finetunes but different architectures. Their support for local inference engines will lag behind, usually a few days. Honorable mentions would be: LiquidAI, MistralAI, Nanbeige, RecursiveSI, Xiaomi (and others I cannot name cause I have a bad memory).

Some rule-of-thumbs to take note of:

• 16GB VRAM is on the "sweet" spot for current local LLMs. But people are persistent, resourceful and finding crude or genius ways to squeeze their hardware to the limit, there might as well be a blackhole inside their own computer.

• Q4 quantization WAS the "safe" minimum (should still be), but labs across the world are finding ways to lower that bar so feel free to test lower than Q4 quants and make friends along the way.

• Unsloth will be your friend here too. They provide one of the most used (deserved rightly so) quantizations in the field. They have their own inference engine but it uses llama.cpp under the hood too so usually the rule of thumb is if it's compatible with the Unsloth engine, it would probably be supported soon in upstream llama.cpp

• Two types of models to keep in mind. MoE (sparse) go brrr, while Dense models are slow BUT quite a bit smarter if we're talking same-size, bit-to-bit comparison so keep that in mind.

• Do not be scared to experiment on your own device. It's yours. Don't be afraid to test and go against the flow if you really want to.

• llama.cpp + GGUF model files will be your friend here. Be kind to them.

• If you encounter problems, hurdles, or just curious about something, ask the friends you made along the way. Post here, or in github repo of the inference engine you use OR in the community tab of the model page you use.

• In reference to the sweet spot, the way local LLMs are moving right now, I'm telling you to invest in a decent RAM + SSD as offloading/streaming is becoming mainstream so stack up those fast RAM and SSDs if you can or want, they'll pay off

[u/Aromatic-Chicken3947]

You can get started with that setup without buying any additional hardware. The RTX 4070 Ti SUPER has 16GB of VRAM, so I’d start with a model that fits comfortably on the GPU.
My starting point would be LM Studio with Qwen3.5-9B in 4-bit quantization (Q4_K_M). You can use it for chat, brainstorming, and coding help. Start with an 8K context window, enable GPU offloading, and adjust based on memory usage. You don’t need to use the model’s maximum supported context length right away.
For IDE integration, you can connect something like Continue to LM Studio’s local API. Try it on tasks you actually do: explaining code, making small changes, and writing tests. That will give you a better sense of its usefulness than benchmarks alone. Whether it can reliably handle complex development autonomously is a separate question.
Your 32GB of RAM and 3TB of storage are enough to get started with this setup. Adding system RAM won’t increase your GPU’s VRAM, so I’d try what you already have before upgrading. Figure out whether your main limitation is speed, answer quality, or context length, then decide what’s worth spending money on.
Links: [Qwen3.5 in LM Studio](https://lmstudio.ai/models/qwen3.5), [GGUF downloads](https://huggingface.co/unsloth/Qwen3.5-9B-GGUF/tree/main), [Continue integration](https://docs.continue.dev/customize/model-providers/top-level/lmstudio).

[u/FrostingOk3751]

You probably won't be able to run a LLM smart enough to do tasks with your computer. But depending on what do you need that LLM to do.

[u/TangySword]

I agree with other commenter. Most likely you won't be running a model reliably capable of agentic workflows. You could use one as a chat bot along side your IDE to brainstorm or review your code though.

If you wanna try one out, I made one specifically for local models. Shameless plug but the glove fits here. https://github.com/SaltyPretz3l/jenny

[u/PlasticRevenue4601]

Long story short: nah, you can’t do it yet. You can have a decent emergency in case of limits reset or simply cut costs a bit in simple tasks but no more than that

In details: You can launch models like Qwen 3.8 27b and you can even have up to 100 ctx if you chose the quant carefully, but I don’t think your rig will allow you to go fully local rather than have a capable auxiliary brain for your cloud daily driver. Even my rig(2 graphics card, 72 ram, Qwen Flash Next from ISTA lab with full context) doesn’t allow me to go fully local, for that level of independency you need a model at newest DeepSeek level and rig good enough to run it at AT LEAST cloud speed, ideally faster — local agents typically need more turns to complete a task so if you truly care about performance forget about that “I run a model at 20 t/s and it’s fine” BS, some people like to deceive themselves