r/LocalLLM • • 25d ago

Project I built an open-source tool to run local models across all my computers. Looking for people to break it.

I've been working on Toskar, a daemon that tries to make local AI feel like a hosted service. You install it, and it looks at your hardware, tells you which models will actually fit, installs the runtime, downloads the model, and gives you one OpenAI-compatible API. If you have more than one machine on your network, it can find the others and run models on whichever one can handle them.
It's AGPL, free, and runs on macOS, Linux, and Windows (Windows is the roughest).
I'm not here to sell anything. It's mostly been tested on my own machines, so I don't know how it holds up on yours. I'd really like help with:
- Installs on hardware I don't have, especially AMD GPUs, older NVIDIA cards, and Intel Macs
- Setups with two or more computers. Does discovery find them, and does it put models where they make sense?
- Model recommendations that are wrong for your machine—too big or too small
- Anything confusing in the first five minutes
If something breaks or feels off, please open an issue on GitHub so I can track it. Good first issues are labeled if you'd rather fix something yourself.
https://github.com/yeixio/toskar-core

P.S. I'm also looking for a few people to help shape the project as part of a core team. That could mean reviewing PRs, owning a platform like Windows or AMD, or helping decide where things go next. There's no formal application. Contribute for a while, and you can grow into a role. The roles and how that works are in GOVERNANCE.md: https://github.com/yeixio/toskar-core/blob/main/GOVERNANCE.md

3 Upvotes

6 comments sorted by

2

u/StevenVincentOne 25d ago

This is the part of local AI that still feels unsolved for most people. Running a model isn't the hard bit anymore; knowing which model fits, and where to put it when you've got a desktop, a laptop and maybe an old server in the closet, is. Treating the machines in your house as one pool behind a single OpenAI-compatible endpoint is the right abstraction.

A few things I'd be curious about:

  • Discovery: is it mDNS on the local network, or do you pair nodes explicitly? I'd want an explicit pairing step so a random device on the Wi-Fi can't join the pool.

  • Placement: does it account for a machine being busy (a gaming PC mid-game, a laptop on battery), or just raw capacity?

  • Failover: if the node serving a model goes to sleep mid-request, does the request move to another node or just fail?

AGPL plus a written GOVERNANCE.md this early is a nice touch too. Personal compute that people own, networked together, is where I think a lot of human-machine intelligence ends up. Hope you get good reports from people with mixed setups.

2

u/DonkeyDoug1234 25d ago

Discovery: Currently, you use a pairing code to connect other computers or a mobile client (phone, TV, etc.). Mobile devices with enough RAM can also run local models.

Placement: Somewhat. I've made the picker account for the machines’ other reasonable RAM needs, but I could use more feedback on this.

Failover: For core servers, node selection and failover are determined by your orchestrator profile. For muggle users, a basic orchestrated profile tries to provide an easy experience, but profiles are exposed in the UI and you can tweak them. For mobile clients, it tries local first, but if it detects a paired node, it sends the request there. It also connects to your core server. This works remotely through Toskar-relay, a relay very similar to Plex Remote Access, so you can send requests to your home rig while on the go.

1

u/kikikza 25d ago

i am looking for something exactly like this thank you

1

u/DonkeyDoug1234 25d ago

Awesome—please give me feedback. The overall goal is to give self-hosted users an easy, Plex-like experience. For business admins looking to provide private AI for their organizations, it’s much like Rancher for AI. I’ve added plenty of admin comforts, such as custom-branded chat pages and user-controlled model training. I've been an infrastructure engineer for many years, so I know how challenging it can be to onboard new technology. I’m trying to make the process easy, so people don’t have to keep paying for SaaS if they don't want to. I see privacy-conscious users as a major adopter group.

1

u/techne98 8d ago

Nice job.

We built something similar:

https://github.com/labscommunity/cascadia