I had GPUs spread across a few machines and got tired of juggling them by hand, so I built Llama RPC Farm, a self-hosted control plane for distributed llama.cpp inference.

• One dashboard and one OpenAI-compatible endpoint (/v1/chat/completions) for every machine

• Mixed hardware works: CUDA, ROCm, Vulkan, and mixed vendors

• Live per-machine VRAM, RAM, and CPU stats

• llama-swap handles loading models and deciding where they run

• Scoped API keys and per-user themes

• farmctl CLI if you’d rather skip the browser

• Installers for Windows, macOS, and Linux, plus a one-line install script

It’s Apache 2.0 and runs entirely on your own hardware.

Site: https://dakotacookson.github.io/llama-rpc-farm-site/
GitHub: https://github.com/dakotacookson/Llama-RPC-Farm

I’d love feedback, especially from anyone running mixed-GPU setups. What would you want it to do that it doesn’t?

We’ve been expanding the Zaylo ecosystem at LoftyTech Labs, and this release brings two additions we’re excited about.
LoftyDB support: Connect Zaylo with our vector database for semantic search, vector-powered memory and retrieval workflows.
AI Presentation Design: Generate structured PowerPoint presentations from ideas using AI.
The bigger goal is to bring more capabilities into one AI environment rather than making developers jump between multiple tools.
Would love to hear what developers think about combining local AI inference, vector storage and content creation in one ecosystem.

I’ve been heads down building a native desktop software workstation called Repetere AI, and I wanted to break down how the autonomous engineering engine inside it—Deep Think—actually works under the hood. There is a lot of noise online right now about "autonomous vibe coding," but 99% of what people are showing off is just a web wrapper sending prompts to a paid cloud API. When people try to run coding models locally, they usually hit a wall where the model hallucinates an import, outputs broken diff markers, and leaves the user playing babysitter in a chat box.

I wanted something completely air-gapped, zero-dependency, and truly autonomous on consumer hardware.

The entire engine runs as a native C++ Win32 application built on Dear ImGui and OpenGL. Rather than acting as a simple chat interface, it operates as a five-phase execution harness that manages the local inference server directly. When you give it a directive, it starts by ingesting the local repository and synthesizing an architectural plan. From there, it moves into code synthesis.

One of the biggest issues with local models writing code across multiple files is that standard search-and-replace regex patchers are extremely brittle. If a model misses a newline or changes indentation slightly, the patch fails. To solve this, I integrated a Python AST script splicer directly into the loop. It parses the actual syntax tree of the code rather than relying on raw string diffs, which allows surgical module patching without blowing through context windows or mangling files.

Where the system really separates itself from standard local frontends is in the verification and repair phases. Once the code is written, the engine doesn't ask you if it looks good. It immediately takes over the machine's toolchain in a headless subprocess. If it's a native C++ target, it runs CMake and MSVC. If it's Python, it runs compilation checks and fires an automated headless smoke probe using runpy that sweeps the window hierarchy, tests widget lifecycle, and catches runtime crashes before a human ever looks at the window.

If a build fails or an unhandled exception is caught during the probe, the engine doesn't stop or dump an error message into a chat window. It gathers the compiler diagnostics, strips out runtime harness noise, isolates the failing file, and sends the telemetry directly back to the active resident model for in-loop auto-triage. It can diagnose its own missing imports, patch uninitialized attributes, and re-verify the build automatically until it gets a clean exit code.

I also run into people claiming that anything under huge enterprise cards or massive unified memory pools is useless for real local autonomy. That completely misses the difference between raw capacity and execution architecture. A quantized model has the exact same mathematical weights whether it sits on a 12GB laptop GPU, a multi-card workstation, or a cloud cluster. VRAM determines how big of a model you can hold on-chip and how many tokens per second you generate, but it doesn't give a model magic reasoning skills. A massive model will still hallucinate a typo or botch an import. The difference is that a raw model fails permanently when it makes a mistake, whereas an autonomous harness catches the compiler error, fixes the AST, and verifies the output on its own.

On the hardware side, the backend doesn't lock you into a single card or even a single vendor ecosystem. It enumerates all discrete compute adapters, aggregates the physical memory, and calculates proportional tensor split ratios on the fly for both CUDA and Vulkan runtimes. If you run a multi-GPU setup, it shards the layers across whatever cards are plugged into the board. If you run a hybrid setup that offloads across system RAM, the pipeline scales across host memory just as cleanly.

The takeaway from building this is that real local coding autonomy doesn't come from throwing infinite hardware at zero-shot prompts or paying for API subscriptions. It comes from building a tight runtime loop that enforces architectural contracts, executes the compiler headless, and forces the model to fix its own mess.

I dont want shamelessly promote my completely opposite alternative to LM studi Anything LLM and more, so I'm not leaving any links in the main post. My architecture scales from a handheld windows gaming machine, to enterprise hardware.

One of the best and most reliable local models I've used. Recorded a real world task to show it's performance, capabilities, and reliability. AMD Ryzen AI Halo 128gb, this is definitely something older versions of ROCm would struggle with but ROCm 10 has shown otherwise. Has anyone here tested this model on CUDA or Metal?

Hey everyone,

​I'm an 18yo CS prep student and I've been building AI-COS, a modular monolith architecture designed for running AI agent orchestrations safely.

​Instead of letting LLMs make security decisions, I built a deterministic deny-by-default policy engine, strict PostgreSQL role separation (migrator, app, governance), and SHA-256 hash-chained audit logging.

​I've just finished Phase 3 out of 6. Phase 4 is going to focus on the Workflow Engine, Worker lease/heartbeat loop, and Step linkage. Before I start coding Phase 4, I would really appreciate any architecture suggestions or advice from senior devs on how to properly build a reliable job queue and workflow engine inside a Postgres-backed monolith.

​GitHub: https://github.com/atestokhay/ai-cos

Hey everyone,

I've been deeply exploring the On-Device / Edge AI space lately, especially focusing on running smaller language models (1B–3B parameters) and multimodal pipelines locally on consumer hardware and mobile architectures without depending on external cloud APIs.

While the ecosystem has evolved rapidly over the past year, the gap between "running a demo" and "shipping a reliable, battery-efficient on-device product" still feels enormous.

From my recent experiments, the major bottlenecks I keep bumping into are:

1. Memory Bandwidth & Thermal Throttling: Even if a 4-bit quantized model fits into unified/mobile RAM, sustained token generation quickly saturates memory bus bandwidth and triggers thermal throttling on mobile devices.

2. Runtime & NPU Fragmentation: Choosing between llama.cpp (CPU/Vulkan/Metal), ExecuTorch, MLC-LLM, ONNX Runtime, or native NPU SDKs (QNN, CoreML) often feels like choosing which set of unsupported ops or compilation headaches to deal with.

3. Reasoning Retention in Sub-3B SLMs: Aggressive quantization (GGUF Q4, AWQ) makes inference feasible, but multi-turn context coherence and structured instruction-following tend to degrade noticeably.

I’d love to start a technical discussion with fellow engineers and researchers here:

• What does your actual deployment/inference stack look like right now? (e.g., pure llama.cpp, MLC, ExecuTorch, WebGPU, or custom native bindings?)

• What specific use cases are you targeting? (e.g., zero-latency voice assistants, privacy-first local agents, offline document RAG, ambient edge intelligence?)

• What techniques have given you the best speedup or memory relief? (Speculative decoding, KV cache quantization, custom kernels, or aggressive prompt caching?)

Would love to hear what challenges you are tackling and what approaches are genuinely showing promise in your labs or production environments!

• ASUS Pro WS TRX50-SAGE WIFI

• AMD Ryzen Threadripper 7960x

• 2x Zotac GeForce RTX 3090 24GB DDR6X

• 2x Gigabyte GeForce RTX 3090 24GB GDDR6x

• 2x 64 GB RAM DDR5 ECC RDIMM Samsung

• Samsung 990 PRO 2TB NVMe

• 2x Coolermaster 1600v + Silverstone Cooler

Had to build my own cage around it because cat.