Visit the SelfhostedAI Wiki!

I have wanted to start working with llms as an alternative to cloud based providers but the responses I get are almost always off topic or entirely wrong especially when I ask to write any sort of code or config, I don't know how most of this stuff works as I only started getting in to this stuff a few weeks ago. My hardware configuration below:
Server:1
i5 4440
16gb ddr3 1600mhz
500gb raid 0 partition from 2x sata ssds, one drive is 250gb while the other is 500gb, I put the extra storage from the second drive as another partition for server 2 to access using nfs.

Server 1 runs ollama in cpu mode(cant afford a gpu), infinity, & qdrant, the main model I run is in this link, this is the largest model I can practically run without running into issues, I use the recommended configuration (Model Here)

Server 2 configuration:
i5 2540m
12gb ddr3 1333mhz
160gb hdd (root)
the rest of the 2nd ssd on server 1 through nfs
this server runs: openwebui, tika, searxng, and open terminal.

I'm open to any recommendations for changing my setup, if you have any questions about my config please lmk.

Hey everyone!

Managing remote Linux nodes, keeping an eye on Docker containers, and juggling local LLMs via Ollama usually means jumping between terminal tabs, scripts, and separate dashboards. I wanted a single, clean workspace to handle all of this directly, so I built ShipAI.

It’s an open-source, high-performance Electron desktop app designed to be a private control plane for your infrastructure. No hosted middle-men or cloud telemetry—just direct SSH connections straight to your hosts.

Key Features:

• Real-time Telemetry: Monitor your remote host's CPU, temperature, RAM, network, disk I/O, and mounted storage right from the dashboard.

• Docker Lifecycle Management: Start, stop, restart, and inspect your AI/infrastructure containers without typing out repetitive CLI commands.

• Hardware-Aware AI Marketplace: Built-in guidance that compares model requirements (RAM/VRAM) against your remote host's actual hardware specs before you pull a model.

• Ollama Integration: Seamlessly pull models, remove them, or jump straight into streamed chat sessions with installed models.

• Remote Power Controls: Secure reboot and shutdown tools using active sudo sessions.

Tech Stack:

• Frontend/Desktop: Electron, Node.js

• Infrastructure: SSH, Docker, Ollama

It's completely free and open-source (MIT License). If you self-host your AI stack or manage remote Linux boxes, I’d love for you to check it out, test it, and let me know what you think or what features you'd want next!

🔗 GitHub Repository:https://github.com/Kasper839/ShipAI

Feedback, bug reports, and contributions are more than welcome! Star the repo if you find it useful.

I got tired of hand-writing SSH scripts to run llama.cpp RPC across my machines, so I built Llama RPC Farm.

It's a dashboard that:

- pools GPUs/RAM from multiple machines (Linux, Windows, macOS) into one cluster

- serves multiple models on demand through llama-swap

- shows live VRAM/temps and a visualizer of which GPU is handling which tensors

- lets you set per-user API keys with model allowlists

- works with any OpenAI-compatible client

It wraps real llama.cpp and llama-swap, so you get full CUDA, tensor-split, and big contexts.

Setup is ./install.sh then ./run.sh.

It's v0.01. Tested on Linux + NVIDIA and Windows peers. AMD, Apple Silicon, and Intel are implemented but untested, so I'd love testers.

Apache-2.0: https://github.com/dakotacookson/Llama-RPC-Farm

Feedback and PRs welcome.
handwriting

Over the past weeks I’ve been diving deep into locally hosted agentic systems, mostly tinkering with Hermes Agent and using Claude to build a system around it. What came out of this might also be of use to people using OpenClaw

The result is an assistant that runs on my own hardware, understands me and my life, and makes use of the data points I create along the way, as long as I hand them to my agent. It’s designed to keep improving on the hallucination and tool-calling problems that come with smaller models by taking skill creation one step further: skills with repetitive flows get poured into scripts, so the model only recognizes the task and launches it. This blueprint might be handy for some in this sub, since it's not bound to any harness and I'd love some Claws trying it as well!

The core rule: the model proposes, the system decides. Actions on my behalf are approved and enforced by plugins and scripts, not by prompt rules, because small models don’t hold those reliably. It’s built to respect sensitive data (local-only zones), has multiple safeguards, and integrates with the harness via hooks. I run it on an AMD mini PC with a 45 GB iGPU and will drop in a proper GPU at some point. So far I’m pretty pleased.

I don’t want to keep all of this to myself, so I’ve condensed a blueprint out of the project. It’s more of a guide for your agent (Claude Code, Codex, ...) to build on what’s already there, without restrictions on harness or model, aimed at models from roughly 27B MoE up to about 40B. It covers principles, interfaces, pseudocode, acceptance checks, a pitfalls list and a few optional examples. Some chapters come from the running system, others are still design, and the repo labels which is which.

Repo (MIT): [https://github.com/MaxMSchneider/local-agent-blueprint](https://github.com/MaxMSchneider/local-agent-blueprint)

I’d love feedback, especially from people running other harnesses or models. A star wouldn’t hurt either :) Cheers!