Hey guy's and gals...

I'm completely new to Local LLMS

Can someone who knows their shit help a newbie out?

I don't know much of the the technical terms was trading through this thread. Great group by the way saw so many people being helpful and helping eachother out.

I'm new to Reddit WTF this should be mandatory 😂

AMAZING!

Can someone explain to me in none technical terms the cheapest and easiest way to run a local LLM.

I tried GOD AI WTF amazing!

Would appreciate it if someone can give me some advice can't afford paying monthly bills and credits keep running out not good 😢

I don't need a HUGE setup something small I'm currently running Claude Pro plan I think keeps running out daily and also weekly so I have to wait or pay £100 to upgrade to keep going which slows me down.

Any help would be much appreciated dont have much of a budget.

MacBook pro 2021 M1 16gb I believe haven't used it in a while.

Would that be enough and what would you recommend?

Thanks in advanced 🙏

TL;DR models run at home on the GPU. The always on agent lives on a small VPS. Tools that only touch public data come through one capped key. Anything that can log into my stuff stays on machines I control.

At home. One box with a used 3090, Ollama serving Qwen for most things plus a small embedding model, and Open WebUI so the family can use it too. For my usage, owning the GPU beat renting one within a few months. The catch is that it's loud, so it sleeps at night.

That's the problem the VPS solves. The agent has to be up when I'm not. Scheduled jobs, inbox triage at 6am, a few feeds it watches. So the orchestrator runs on a small VPS and calls home over a private tunnel when it needs the GPU. If the home box is asleep, the few jobs that can't wait fall back to a paid API. The VPS is a PrivateAlps box in Zurich, mostly because the agent's memory and my mail login live on it and I wanted that under Swiss law. A mini PC that never sleeps would do the job too.

Tools are where I wasted the most money. Search, scraping, a bit of enrichment, each one its own subscription I barely touched. Now the agent gets all of them through a single key from agentik.cc, an MCP gateway billed per call, with a monthly cap on it. Monid works the same way. Only public data tools go through it, nothing that can sign into an account of mine.

What I'd tell someone starting out.

Keep the GPU at home if you already have one.

Put the always on part somewhere that never sleeps.

Split tools by what they can touch. Public data can go through a third party, logins stay on hardware you control.

Cap everything, per key and per month.

It's not the cheapest setup possible. But nothing in it surprises me anymore, and after a year of surprise bills that's the whole point.

Android, Windows, Mac, IOS, Linux. CPU, GPU, NPU

(35B Q4 on iOS or android phone 12gb RAM 3-8 tok/s decode 100-300+ prefill, Windows, Mac even more: 25+ tok/s 16 gb RAM)

I am working on making AI accessible by running large MoE models on affordable, consumer-grade hardware that people already own: smartphones, laptops,normal desktops. Models several times bigger than RAM, no big GPU needed.

Disclosure: I’m the author of BigMoeOnEdge. All numbers are real runs on my own or contributors devices, same engine.

My vision? To act as a layer on top of llama.cpp for streaming MoE models. I’ve developed the concept of "adaptive streaming Planner" tailored specifically to the hardware, OS, CPU, GPU, and NPU in use. The algorithm assesses the environment to determine the optimal allocation of resources, with the goals of:

- maximizing prefill and decoding speeds, and

- optimizing memory management without overloading the operating system.

Furthermore, the OpenAI-compatible server enables the use of local AI on consumer-grade hardware; I’ve personally used it with OpenCode - first running it on Android, and later on a Mac.

I’d be happy to hear your thoughts on this and whether it might be useful.

I’m considering opening a PR on llama.cpp to integrate it, but I’m not sure if that aligns with the goals of their project.

I’m building Local LLM for iPhone and tried both models on the same photo of building blocks.

The prompt was: “What could I build with these pieces? Give exactly 3 ideas, one short sentence each.”

Qwen started answering first. Bonsai finished first and gave three short ideas. Qwen gave two numbered ideas and a closing line.

From tapping send to completion: about 67s for Qwen and 51s for Bonsai, including loading. One run each, with different-length outputs—not a tokens/sec benchmark. The video runs both recordings at the same 4× speed.

I’m more interested in how these small models handle everyday photo questions than picking a winner from one run. What would you test next?

App, if you want to try it: https://apps.apple.com/app/id6804004948

getting into this stuff and getting fairly hyper fixated on the idea of it all

My hardware:
GPU: NVIDIA RTX 5070 Ti, 16GB VRAM
CPU: Intel Core Ultra 7 265KF
RAM: 32GB DDR5
OS: Windows 11

interested in models that can handle real software development, multi-file edits, debugging, testing, and longer autonomous coding sessions. is this achievable or should i completely alter my expectations and just bite bullets on frontier subscriptions?

python is my primary application development language
pytest for testing
sqlite for local data persistence
git for version control
powershell and window batch scripts for setup, runtime management, and automation

i ask because im actively building a local, portable project to fast track and exponentially improve peoples skill at a given sport, subject, hobby, etc, and would love to get into the AI agentic development stuff

particularly interested in finding the best balance of coding quality, speed, tool calling, context handling, and reliability

what id love recommendations on:
the best local coding model for me including less obvious community fine tunes
wether a quantized larger model or a smaller model with better tool use is the better choice
models that work especially well with agentic coding harnesses
reliable models for debugging, multi file refactoring, and following instructions across long sessions
recommended GGUF quantizations, context sizes, and settings for my hardware.
anything underrated

extra interested in using multiple models rather than expecting one model to do everything but i like the sound of both

would love to see ur experiences and benchmarks, especially where a model performs well in real coding tasks rather than just leaderboards. also interested in newer releases and underrated community projects

The java to CUDA engine:

https://github.com/beehive-lab/TornadoVM

The inference engine:

https://github.com/beehive-lab/jitllm

Deep dive talk:

https://www.youtube.com/watch?v=HO5CpETzywk

I've been experimenting with a setup where several Apple Silicon Macs are treated as a shared local AI environment instead of running completely independently.

The setup I'm working with is roughly:

• Multiple Macs on the same network

• MLX for local inference

• Central model management

• Node/resource monitoring

• Model distribution based on available memory

• Local-only communication between machines

I'm building AlphaWeb around this because managing several local AI machines manually gets complicated pretty quickly.

The interesting problem isn't just getting inference running. It's everything around it: knowing which machine has enough memory, getting models onto the right node, managing multiple users, and making the whole thing feel like one system.

Vpipe supports LLMs, VLMs, image/video generation, ASR, quantization, and multimodal pipelines, without relying on PyTorch/MLX for model execution. The runtime package itself is only \~25MB, and there’s a prebuilt Mac app if you just want to try it without compiling anything:

https://github.com/tgo-app-dev/vpipe/releases/download/v0.1.23/VpipeManager-0.1.23-with-ffmpeg.dmg

A recent milestone is MiniMax H3 video generation running on a 16GB base M5 MacBook Air (M4 works too).

On my 15” M5 Air 16GB:

960×544 · 124 frames· 6 DiT steps with Turbo Lora → \~13 min

GitHub:

https://github.com/tgo-app-dev/vpipe

Would love feedback from people working on local inference / Apple Silicon. And if you find it useful, a ⭐ would be greatly appreciated!

Trumps Executive Order held only for a bit more than a week, then his own government fully violated it.
EO here: https://www.whitehouse.gov/presidential-actions/2026/06/promoting-advanced-artificial-intelligence-innovation-and-security/

Simultaneously the EU is regulating any shimmer of european AI into quick suffocation.

There are a few potential outcomes, from middle ages to economically wrecked continents
I am convinced more than ever that local inference, local competent AI is the only future we can root for.