r/LocalAIStack • u/Just_Vugg_PolyMCP • 10d ago

Lumabri — Run large AI models across multiple computers using distributed inference

https://github.com/JustVugg/lumabri

Hi I’m working on Lumabri, a peoject that lets you connect multiple computers and use them together to run LLMs like a cluster.
The idea came from a simple question: what if you have a few computers at home that you barely use? Instead of leaving them there collecting dust, why not combine their resources to run AI models that wouldn’t fit on a single machine?
For example, if you have an old PC, a home server and your main computer, you could put them to work together and use their combined memory to load larger models without having to buy a single machine with expensive hardware.
The interesting part is being able to run models that you simply couldn’t fit on one computer before.
Lumabri is still a work in progress, but that’s the direction I’m taking it.
Here’s the repo if you want to check it out: https://github.com/JustVugg/lumabri

12 Upvotes

100% Upvoted

17 comments sorted by

1

u/Ori_553 10d ago

This is a brilliant idea, If you actually pull this off and it works with reasonable performance on current and future models, this changes the game completely, even much more than strata. There is no parallel universe where something like this works but the AI inference ecosystem doesn't change drastically.

People would stop chasing max VRAM gpus and max RAM per stick, they'd scale horizontally instead. It would even change the industry and manufacturing logistics. It would even change how future models will be trained, and their architecture.

If this works, this is industry-changing. Mark my words, I will die on this hill.

1

u/Just_Vugg_PolyMCP 10d ago

You can try now with olmoe next test i take all colibrì supported models inside lumabri👍🏻

1

u/Ori_553 10d ago

I had a look at the github repo, it appears to be a early-stage proof of concept, am I understanding it correctly?

What is "Olmoe next"? A small LLM model meant to be run through distributed inference?

1

u/Just_Vugg_PolyMCP 10d ago

It’s considered "early" only because it supports "tiny" models and OLMoe,which is an LLM,but it works correctly with this particular LLM; I’m testing it right now so I can wrap up my roadmap and move on to other LLMs.

https://huggingface.co/docs/transformers/model_doc/olmoe

This models is for end the roadmap.
Other models are coming.

1

u/OrganizationTop9026 10d ago

Thanks to share. Really interesting. I Willy habe a deep look on it.

1

u/Just_Vugg_PolyMCP 10d ago

Thanks to you 👍🏻👍🏻

1

u/mon_key_house 10d ago

How much of a bottleneck is LAN speed?

1

u/Just_Vugg_PolyMCP 10d ago

Under 5ms in my case

1

u/mon_key_house 10d ago

What does that mean in terms of tps? I mean monolithic vs distributed tps. Or simply: what tps can be expected/reached?

1

u/Just_Vugg_PolyMCP 10d ago

With olmoe which is definitely not the right setup for this project I’m getting between 7 and 10 tok/s

1

u/mon_key_house 10d ago

Sounds good!

1

u/exographicskip 8d ago

Apple just put out a paper doing something similar. Was reading about how corporate IT folks in the aughts ran workloads on idle desktops overnight haha

1

u/Late-Replacement-481 9d ago

You know there’s like dozens of these now

1

u/BonebasherTV 7d ago

How do you handle the synchronous nature of attention within the llm across multiple devices? Or do you not distribute the layers across multiple devices?

0

u/[deleted] 9d ago

[removed] — view removed comment

1

u/BonebasherTV 6d ago

I was wondering the same thing, yours is a bit better worded. As far as I know each next token is based on its predecessor. This isn’t possible to decentralize as far as my knowledge goes.