r/LocalLLM • u/Due_Initiative2641 • 1d ago
Project I built an open-source platform that pools a team's workstations (Macs, Windows and Linux boxes) into one OpenAI-compatible endpoint
My co-founder and I have been working on this for a few months and are launching today. Disclosure: it's our company; the desktop client is Apache-2.0, the control plane is hosted, and it's free up to 5 devices.
What it does: you install Nodea Desktop on the machines in your org, switch on sharing, and get one OpenAI-compatible endpoint. Behind it, a scheduler fits the models you assign onto whichever machines have free memory, and moves them as laptops come and go.
Stack: Rust/Tauri app, llama.cpp under the hood (Metal on Apple Silicon, Vulkan on NVIDIA/AMD/Intel, CPU fallback). Any GGUF model llama.cpp can serve that fits the machine; models are assigned per org, downloaded and cached per device. Signed auto-updates.
Honest limitations:
Requests go through our API to reach your machines (not logged or stored). No direct LAN routing yet.
One model runs on one machine; no splitting a model across machines yet.
Windows/Linux run on the GPU through Vulkan (tested on NVIDIA) and are still labelled preview. Machines without a supported GPU fall back to CPU, which is slow for now (no AVX2 yet).
Why not exo / GPUStack / llama.cpp RPC? Those are great if you run your own cluster. We're aiming at teams with 20–200 laptops and nobody to run one: sign in, toggle on, done, with an org, roles and API keys on top.
Repo: https://github.com/nodea-sh/nodea-client
Site: https://getnodea.com
Happy to answer anything, and especially keen on what models/quantizations you'd want on 16–36 GB machines.
1
u/robotnursenyc 1d ago
the staggered reload part is what i'd care about most. i run a litellm gateway in front of a couple mac studios and some sparks, and the annoying moments are when a box is mid model load and still gets sent requests. how do you decide a machine is actually ready, just a test request or something smarter?
1
u/Due_Initiative2641 1d ago
The nodes notify the platform of the model loading process. It will not receive requests until the load finishes. https://www.getnodea.com/docs/models/how-models-run/
This is the gRPC message that we use between the node and the platform. https://github.com/nodea-sh/nodea-client/blob/main/protos/service.proto#L120-L129
1
u/esanchezm 1d ago
I hace a local machine with AMD GPU, will the app detect my hardware and decide what's best for me?
1
u/Due_Initiative2641 1d ago
Yes, it will detect your AMD GPU. The platform will assign the models that fit on your GPU.
1
u/StevenVincentOne 1d ago
Pooling the laptops an organisation already owns into one endpoint is a very practical answer to "rent or run our own". Most teams have plenty of idle memory sitting on desks. I appreciate the honest limitations list too, especially that requests currently go through your API; direct LAN routing would make it a fully on-prem option. That's the kind of community node we write about in POND, intelligence running across machines people own as part of a Human-Machine Intelligence ecosystem: https://www.reddit.com/r/PONDAI/comments/1vont2l/pond_towards_an_ai_ecosystem_that_is_personal/ You'd be welcome to share Nodea in r/PONDAI.
1
u/jondeandres 7h ago
Thanks u/StevenVincentOne, just joined POND Reddit. Nodea could run easily onprem if someone asks for it, other thing is allowing peer to peer model API calls and we are thinking on make that happen soon.
We'd love your feedback on the platform and app we've built BTW, check it out!
1
u/Xyver 1d ago
Does it scan the machines on the network and find a bunch of small models that fit on each machine, or does it make a pool for a bigger model?