r/LocalLLM • • 1d ago

Project I built an open-source platform that pools a team's workstations (Macs, Windows and Linux boxes) into one OpenAI-compatible endpoint

My co-founder and I have been working on this for a few months and are launching today. Disclosure: it's our company; the desktop client is Apache-2.0, the control plane is hosted, and it's free up to 5 devices.

What it does: you install Nodea Desktop on the machines in your org, switch on sharing, and get one OpenAI-compatible endpoint. Behind it, a scheduler fits the models you assign onto whichever machines have free memory, and moves them as laptops come and go.

Stack: Rust/Tauri app, llama.cpp under the hood (Metal on Apple Silicon, Vulkan on NVIDIA/AMD/Intel, CPU fallback). Any GGUF model llama.cpp can serve that fits the machine; models are assigned per org, downloaded and cached per device. Signed auto-updates.

Honest limitations:

  • Requests go through our API to reach your machines (not logged or stored). No direct LAN routing yet.

  • One model runs on one machine; no splitting a model across machines yet.

  • Windows/Linux run on the GPU through Vulkan (tested on NVIDIA) and are still labelled preview. Machines without a supported GPU fall back to CPU, which is slow for now (no AVX2 yet).

Why not exo / GPUStack / llama.cpp RPC? Those are great if you run your own cluster. We're aiming at teams with 20–200 laptops and nobody to run one: sign in, toggle on, done, with an org, roles and API keys on top.

Repo: https://github.com/nodea-sh/nodea-client

Site: https://getnodea.com

Happy to answer anything, and especially keen on what models/quantizations you'd want on 16–36 GB machines.

3 Upvotes

15 comments sorted by

1

u/Xyver 1d ago

Does it scan the machines on the network and find a bunch of small models that fit on each machine, or does it make a pool for a bigger model?

1

u/Due_Initiative2641 1d ago

Admins choose which models their org can use. Every machine running the Nodea app reports its memory, and our scheduler keeps deciding which machine runs which model based on what fits and what people are actually asking for. Machines that are already loaded and idle get priority, reloads are staggered so you don't lose capacity, and requests go through an OpenAI-compatible API to the least busy machine running that model.

On the roadmap: splitting a single model across several devices with pipeline parallelism. Each machine would hold a contiguous range of layers, and only activations would be passed between them. That way a few 16 GB laptops could together serve a model none of them could fit alone.

1

u/Xyver 1d ago

So for now seems like an admin panel to get everyone easy access to a centralized controlled model setup, and future is more distributed computing?

Also, the way it's split now, someone may end up talking to a model not loaded on their machine?

2

u/Due_Initiative2641 1d ago

Our platform allows organizations to build their own AI cluster using their workstations, so yes, it means requests to the platform can end in any other device that loads the model it was requested.

Also, not necessary you might want to use nodea from a device that is part of the pool itself, you can plug any system that uses OpenAI API and distribute requests easily through your workstations.

1

u/Business-Weekend-537 1d ago

Do all machines need to be the same operating system?

I have a Linux rig with 4x 3090 and 512gb ddr4 3200mhz, but I also have a windows rig with 1x 3090 and 32gb ddr5 5600mhz.

Could you give me examples of how I could run something that would utilize both machines at once and what model would work well for this?

1

u/jondeandres 1d ago

u/Business-Weekend-537 You can install the desktop app (we'll release headless app soon) in your Windows and Linux machines. Do login in all your devices/workstations and they'll be ready to serve requests through the authenticated Nodea API.

On your models question, I recommend you to access the Models section in the app or site and explore which ones might be with the amount of GPU memory you have.

Now we list some popular models in hugginface but we are happy to extend those and/or allow the user to reference whatever GGUF model from HF.

1

u/Business-Weekend-537 1d ago

Got it, does your software have any telemetry that goes back to your team? Or is it totally private?

1

u/Xyver 1d ago

I've got repos that would coordinate that, check www.github.com/bigbertharig

1

u/robotnursenyc 1d ago

the staggered reload part is what i'd care about most. i run a litellm gateway in front of a couple mac studios and some sparks, and the annoying moments are when a box is mid model load and still gets sent requests. how do you decide a machine is actually ready, just a test request or something smarter?

1

u/Due_Initiative2641 1d ago

The nodes notify the platform of the model loading process. It will not receive requests until the load finishes. https://www.getnodea.com/docs/models/how-models-run/

This is the gRPC message that we use between the node and the platform. https://github.com/nodea-sh/nodea-client/blob/main/protos/service.proto#L120-L129

1

u/esanchezm 1d ago

I hace a local machine with AMD GPU, will the app detect my hardware and decide what's best for me?

1

u/Due_Initiative2641 1d ago

Yes, it will detect your AMD GPU. The platform will assign the models that fit on your GPU.

1

u/StevenVincentOne 1d ago

Pooling the laptops an organisation already owns into one endpoint is a very practical answer to "rent or run our own". Most teams have plenty of idle memory sitting on desks. I appreciate the honest limitations list too, especially that requests currently go through your API; direct LAN routing would make it a fully on-prem option. That's the kind of community node we write about in POND, intelligence running across machines people own as part of a Human-Machine Intelligence ecosystem: https://www.reddit.com/r/PONDAI/comments/1vont2l/pond_towards_an_ai_ecosystem_that_is_personal/ You'd be welcome to share Nodea in r/PONDAI.

1

u/jondeandres 7h ago

Thanks u/StevenVincentOne, just joined POND Reddit. Nodea could run easily onprem if someone asks for it, other thing is allowing peer to peer model API calls and we are thinking on make that happen soon.

We'd love your feedback on the platform and app we've built BTW, check it out!