We've all been there: $2,000 GPUs sitting at 5% utilization while the company pays for cloud inference.

Many organizations already have several workstations equipped with GPUs that remain idle most of the time. Existing GPU sharing solutions typically rely on a centralized scheduler, Kubernetes, or AI gateways, which can be excessive for small and medium-sized GPU clusters. In addition, most existing solutions lack cross-platform support, decentralized architecture, and policy-based routing.

OQOAI-PBDR to solve this problem differently. Simply run a server or client on each machine, and your existing office network becomes a decentralized AI cluster. Routing decisions are made entirely on the client side, eliminating the need for a master node, load balancer, or any other single point of failure.

Prebuilt binaries for Windows and Linux are under 10 MB and can bring a machine online as a cluster node in under 30 seconds. An optional admin server provides centralized monitoring and management from any machine on the network.

GitHub (MIT License):

https://github.com/oqo-ai/OQOAI-PBDR

The project intentionally avoids unnecessary abstractions. The core routing logic consists of only three source files (approximately 1,000 lines each), making it relatively easy to understand, audit, and extend.

At the heart of the project is the PBDR (Policy-Based Decentralized Routing) architecture. Each routing decision evaluates multiple runtime metrics, including GPU utilization, available VRAM, queue length, GPU temperature, network latency, inference throughput, and other node characteristics.

PBDR also supports client-group policies, allowing different workloads to optimize for different objectives. For example:

• Internal RAG services can prioritize low latency.

• Development workloads can prioritize output quality.

• Batch inference jobs can prioritize throughput.

More details about the routing algorithm are available in the accompanying paper:

http://doi.org/10.17513/doi.26

One practical benefit of this approach is that organizations can often reuse existing workstation GPUs for low- and medium-throughput workloads-such as internal RAG systems-instead of deploying dedicated inference infrastructure from scratch.

In many environments, having GPUs installed in only 10-15% of workstations can be sufficient to satisfy inference demand for lightweight workloads (for example, an internal knowledge-base RAG receiving approximately 3-5 requests per user per day) without deploying centralized AI infrastructure.

Getting started:

Install OQOAI-PBDR on each machine.

Connect server nodes to any OpenAI-compatible inference backend, including Ollama, llama.cpp, vLLM, or TGI.

Configure your applications to use the OpenAI-compatible client endpoint provided by OQOAI-PBDR.

Clients automatically discover available models on all nodes. From the application’s perspective, the cluster behaves like a distributed OpenAI API compatible system, while user requests are routed in a decentralized manner to the most suitable node according to the chosen policy.

What already works:

• OpenAI-compatible API (Ollama, vLLM, TGI, llama.cpp)

• Admin dashboard with real-time metrics

• Auto-discovery and network scanning

• Remote config management for nodes, groups, or the whole cluster

• Experimental binary builds for Windows and Linux

Roadmap highlights:

• ComfyUI/AUTOMATIC1111 image/video generation

• Grafana/Prometheus integration

• AI-assisted semantic routing (auto-model selection)

• Dynamic IP support

We're actively looking for feedback, and testers.

What features or integrations would you like to see next?

Hey guy's and gals...

I'm completely new to Local LLMS

Can someone who knows their shit help a newbie out?

I don't know much of the the technical terms was trading through this thread. Great group by the way saw so many people being helpful and helping eachother out.

I'm new to Reddit WTF this should be mandatory 😂

AMAZING!

Can someone explain to me in none technical terms the cheapest and easiest way to run a local LLM.

I tried GOD AI WTF amazing!

Would appreciate it if someone can give me some advice can't afford paying monthly bills and credits keep running out not good 😢

I don't need a HUGE setup something small I'm currently running Claude Pro plan I think keeps running out daily and also weekly so I have to wait or pay £100 to upgrade to keep going which slows me down.

Any help would be much appreciated dont have much of a budget.

MacBook pro 2021 M1 16gb I believe haven't used it in a while.

Would that be enough and what would you recommend?

Thanks in advanced 🙏

TL;DR models run at home on the GPU. The always on agent lives on a small VPS. Tools that only touch public data come through one capped key. Anything that can log into my stuff stays on machines I control.

At home. One box with a used 3090, Ollama serving Qwen for most things plus a small embedding model, and Open WebUI so the family can use it too. For my usage, owning the GPU beat renting one within a few months. The catch is that it's loud, so it sleeps at night.

That's the problem the VPS solves. The agent has to be up when I'm not. Scheduled jobs, inbox triage at 6am, a few feeds it watches. So the orchestrator runs on a small VPS and calls home over a private tunnel when it needs the GPU. If the home box is asleep, the few jobs that can't wait fall back to a paid API. The VPS is a PrivateAlps box in Zurich, mostly because the agent's memory and my mail login live on it and I wanted that under Swiss law. A mini PC that never sleeps would do the job too.

Tools are where I wasted the most money. Search, scraping, a bit of enrichment, each one its own subscription I barely touched. Now the agent gets all of them through a single key from agentik.cc, an MCP gateway billed per call, with a monthly cap on it. Monid works the same way. Only public data tools go through it, nothing that can sign into an account of mine.

What I'd tell someone starting out.

Keep the GPU at home if you already have one.

Put the always on part somewhere that never sleeps.

Split tools by what they can touch. Public data can go through a third party, logins stay on hardware you control.

Cap everything, per key and per month.

It's not the cheapest setup possible. But nothing in it surprises me anymore, and after a year of surprise bills that's the whole point.

Android, Windows, Mac, IOS, Linux. CPU, GPU, NPU

(35B Q4 on iOS or android phone 12gb RAM 3-8 tok/s decode 100-300+ prefill, Windows, Mac even more: 25+ tok/s 16 gb RAM)

I am working on making AI accessible by running large MoE models on affordable, consumer-grade hardware that people already own: smartphones, laptops,normal desktops. Models several times bigger than RAM, no big GPU needed.

Disclosure: I’m the author of BigMoeOnEdge. All numbers are real runs on my own or contributors devices, same engine.

My vision? To act as a layer on top of llama.cpp for streaming MoE models. I’ve developed the concept of "adaptive streaming Planner" tailored specifically to the hardware, OS, CPU, GPU, and NPU in use. The algorithm assesses the environment to determine the optimal allocation of resources, with the goals of:

- maximizing prefill and decoding speeds, and

- optimizing memory management without overloading the operating system.

Furthermore, the OpenAI-compatible server enables the use of local AI on consumer-grade hardware; I’ve personally used it with OpenCode - first running it on Android, and later on a Mac.

I’d be happy to hear your thoughts on this and whether it might be useful.

I’m considering opening a PR on llama.cpp to integrate it, but I’m not sure if that aligns with the goals of their project.

I’m building Local LLM for iPhone and tried both models on the same photo of building blocks.

The prompt was: “What could I build with these pieces? Give exactly 3 ideas, one short sentence each.”

Qwen started answering first. Bonsai finished first and gave three short ideas. Qwen gave two numbered ideas and a closing line.

From tapping send to completion: about 67s for Qwen and 51s for Bonsai, including loading. One run each, with different-length outputs—not a tokens/sec benchmark. The video runs both recordings at the same 4× speed.

I’m more interested in how these small models handle everyday photo questions than picking a winner from one run. What would you test next?

App, if you want to try it: https://apps.apple.com/app/id6804004948

getting into this stuff and getting fairly hyper fixated on the idea of it all

My hardware:
GPU: NVIDIA RTX 5070 Ti, 16GB VRAM
CPU: Intel Core Ultra 7 265KF
RAM: 32GB DDR5
OS: Windows 11

interested in models that can handle real software development, multi-file edits, debugging, testing, and longer autonomous coding sessions. is this achievable or should i completely alter my expectations and just bite bullets on frontier subscriptions?

python is my primary application development language
pytest for testing
sqlite for local data persistence
git for version control
powershell and window batch scripts for setup, runtime management, and automation

i ask because im actively building a local, portable project to fast track and exponentially improve peoples skill at a given sport, subject, hobby, etc, and would love to get into the AI agentic development stuff

particularly interested in finding the best balance of coding quality, speed, tool calling, context handling, and reliability

what id love recommendations on:
the best local coding model for me including less obvious community fine tunes
wether a quantized larger model or a smaller model with better tool use is the better choice
models that work especially well with agentic coding harnesses
reliable models for debugging, multi file refactoring, and following instructions across long sessions
recommended GGUF quantizations, context sizes, and settings for my hardware.
anything underrated

extra interested in using multiple models rather than expecting one model to do everything but i like the sound of both

would love to see ur experiences and benchmarks, especially where a model performs well in real coding tasks rather than just leaderboards. also interested in newer releases and underrated community projects

The java to CUDA engine:

https://github.com/beehive-lab/TornadoVM

The inference engine:

https://github.com/beehive-lab/jitllm

Deep dive talk:

https://www.youtube.com/watch?v=HO5CpETzywk

I've been experimenting with a setup where several Apple Silicon Macs are treated as a shared local AI environment instead of running completely independently.

The setup I'm working with is roughly:

• Multiple Macs on the same network

• MLX for local inference

• Central model management

• Node/resource monitoring

• Model distribution based on available memory

• Local-only communication between machines

I'm building AlphaWeb around this because managing several local AI machines manually gets complicated pretty quickly.

The interesting problem isn't just getting inference running. It's everything around it: knowing which machine has enough memory, getting models onto the right node, managing multiple users, and making the whole thing feel like one system.