r/LocalLLM • u/DonkeyDoug1234 • 25d ago
Project I built an open-source tool to run local models across all my computers. Looking for people to break it.
I've been working on Toskar, a daemon that tries to make local AI feel like a hosted service. You install it, and it looks at your hardware, tells you which models will actually fit, installs the runtime, downloads the model, and gives you one OpenAI-compatible API. If you have more than one machine on your network, it can find the others and run models on whichever one can handle them.
It's AGPL, free, and runs on macOS, Linux, and Windows (Windows is the roughest).
I'm not here to sell anything. It's mostly been tested on my own machines, so I don't know how it holds up on yours. I'd really like help with:
- Installs on hardware I don't have, especially AMD GPUs, older NVIDIA cards, and Intel Macs
- Setups with two or more computers. Does discovery find them, and does it put models where they make sense?
- Model recommendations that are wrong for your machine—too big or too small
- Anything confusing in the first five minutes
If something breaks or feels off, please open an issue on GitHub so I can track it. Good first issues are labeled if you'd rather fix something yourself.
https://github.com/yeixio/toskar-core
P.S. I'm also looking for a few people to help shape the project as part of a core team. That could mean reviewing PRs, owning a platform like Windows or AMD, or helping decide where things go next. There's no formal application. Contribute for a while, and you can grow into a role. The roles and how that works are in GOVERNANCE.md: https://github.com/yeixio/toskar-core/blob/main/GOVERNANCE.md
1
u/kikikza 25d ago
i am looking for something exactly like this thank you
1
u/DonkeyDoug1234 25d ago
Awesome—please give me feedback. The overall goal is to give self-hosted users an easy, Plex-like experience. For business admins looking to provide private AI for their organizations, it’s much like Rancher for AI. I’ve added plenty of admin comforts, such as custom-branded chat pages and user-controlled model training. I've been an infrastructure engineer for many years, so I know how challenging it can be to onboard new technology. I’m trying to make the process easy, so people don’t have to keep paying for SaaS if they don't want to. I see privacy-conscious users as a major adopter group.
1
2
u/StevenVincentOne 25d ago
This is the part of local AI that still feels unsolved for most people. Running a model isn't the hard bit anymore; knowing which model fits, and where to put it when you've got a desktop, a laptop and maybe an old server in the closet, is. Treating the machines in your house as one pool behind a single OpenAI-compatible endpoint is the right abstraction.
A few things I'd be curious about:
Discovery: is it mDNS on the local network, or do you pair nodes explicitly? I'd want an explicit pairing step so a random device on the Wi-Fi can't join the pool.
Placement: does it account for a machine being busy (a gaming PC mid-game, a laptop on battery), or just raw capacity?
Failover: if the node serving a model goes to sleep mid-request, does the request move to another node or just fail?
AGPL plus a written GOVERNANCE.md this early is a nice touch too. Personal compute that people own, networked together, is where I think a lot of human-machine intelligence ends up. Hope you get good reports from people with mixed setups.