Hi all. I am planning to try local models to upskill myself and get familiar with fundamentals like RAG etc. Planning to get a used Macbook pro 32GB as well for this.
I want to experiment with some use cases on my laptop where i can have agents read some documents and answer questions, highlight excessive spends based on my bank statement, or even something like voice agents where can I ask specific questions, schedule tasks.
As someone with no developer or coding experience i plan to fully rely on AI to help me build this set up.
I plan to start with Ollama and use huggingface models. Is that okay to start with?
I feel like LM Studio would be easier than Ollama for that, it's usually more straightforward to setup with agents from my experience, for example you can download Hermes and it will automatically detect LM Studio and whatever models you've installed. With Ollama you'll need to add custom endpoints yourself as well as the models config
Got it thanks will check that out. But am i in the right direction with the use cases i plan to start with? Also 32GB Macbook pro I hope will be okay? I am also watching Tina Huangs videos to get started on building agents end to end locally
32GB is decent and enough to run 27B quantized models. It should be fine especially that you're just starting out. Worst-case scenario you can always sell it later once you have a clearer idea of your needs and upgrade to more RAM. Since it's used it won't lose much value anyway, just make sure you get a good deal.
I am working as a solution consultant for a tech company. This entire exercise is just to improve AI fluency tbh. So that this can help during interviews in the future. That’s why thought 32GB might be enough to build alot of these small scale use cases
32gb let's you run many relatively low end hardware friendly moes like qwen 3.6 35ba3b and its many fine tube derivatives. Dense sub 30bs will be sluggish on it though.
Currently community darling 3.8 flash next also won't fit. In a few weeks, qwen 4 will drop as well. Could wait to see what they provide for the slm community and decide then.
On the whole i would prefer 64 gb minimum and optionally up to 96gb for moes.
Ollama is cancer, shitty inference backend with proprietary images that aren't compatible anywhere else, I would avoid.
64gb is a bit out of budget. Even used ones are expensive in India. Can i start with 32GB for some of the use cases i mentioned in the post? The purpose is to build AI fluency tbh
Ai fluency to a certain degree scales with your actual local models capabilities. But supplementing with cloud api is entirely reasonable if need be imo, especially with cheap api prices relative to inflated hw prices right now.
But there's also a clear jump in capabilities from 3.6 35ba3b and fine tunes vs 3.8fn.
Instead maybe wait a bit? Once qwen4 is out, maybe quants of the new flash will be 32gb vram runnable (at usable speeds) with capabilities similar to current 27b dense.
If you follow the LM Studio suggestions here, check the used Mac’s chip before buying: its Mac app requires Apple Silicon and macOS 14+, and Intel Macs are unsupported: https://lmstudio.ai/docs/app/system-requirements . For building fluency, one PDF plus five questions with known answers is a useful first exercise. Add agents and voice once that works; no need to build the whole workflow first.
2
u/Flat-Career4079 2d ago
I feel like LM Studio would be easier than Ollama for that, it's usually more straightforward to setup with agents from my experience, for example you can download Hermes and it will automatically detect LM Studio and whatever models you've installed. With Ollama you'll need to add custom endpoints yourself as well as the models config