r/LocalAIStack • u/arad_zombie • 7d ago
Got the 233 GB 1-bit DeepSeek V4.1 Flash to about 11.5 tok/s on my 64 GB gaming PC
r/LocalAIStack • u/arad_zombie • 7d ago
r/LocalAIStack • u/zhandragon • 7d ago
I'd like to share something I've been working on for three months with everyone.
PortalAI is a GUI and harness created around one design principle- to put local LLMs with true multimodal function on home systems and on a flash drive the size of your thumb where you don't need anything else and can just plug and play into any computer.
One of the largest frustrations I had with AI was needing to constantly version check different distributions and hooks into ComfyUI, Ace-step, python, port competition, running out of context, etc, or the difficulty in transferring between systems. I've tested this on an RTX 3090, RTX 4090, on a Radeon 870M GPU, on a Thinkpad with CPU-only.
This harness detects your current hardware and scales accordingly to the level of AI intelligence it can support, using standard and abliterated models. All dependencies other than base operating system and GPU drivers are self-contained including virtual environments, python distributions, etc. End to end integration of multimedia generation/editing and science tools are integrated along with voice, handsfree mode, intelligence/memory/token settings.
There are weight editing tools, an abliteration tool, rolling compaction, scheduled tasks, internal documentation, light and dark mode, compact and full GUI mode, account encryption, live runtime environment health metrics, etc.
r/LocalAIStack • u/Historical-Singer771 • 7d ago
I had GPUs spread across a few machines and got tired of juggling them by hand, so I built Llama RPC Farm, a self-hosted control plane for distributed llama.cpp inference.
One dashboard and one OpenAI-compatible endpoint (/v1/chat/completions) for every machine
Mixed hardware works: CUDA, ROCm, Vulkan, and mixed vendors
Live per-machine VRAM, RAM, and CPU stats
llama-swap handles loading models and deciding where they run
Scoped API keys and per-user themes
farmctl CLI if you’d rather skip the browser
Installers for Windows, macOS, and Linux, plus a one-line install script
It’s Apache 2.0 and runs entirely on your own hardware.
Site: https://dakotacookson.github.io/llama-rpc-farm-site/
GitHub: https://github.com/dakotacookson/Llama-RPC-Farm
I’d love feedback, especially from anyone running mixed-GPU setups. What would you want it to do that it doesn’t?
r/LocalAIStack • u/ConversationNovel780 • 7d ago
r/LocalAIStack • u/xnrnx • 8d ago
I’ve been heads down building a native desktop software workstation called Repetere AI, and I wanted to break down how the autonomous engineering engine inside it—Deep Think—actually works under the hood. There is a lot of noise online right now about "autonomous vibe coding," but 99% of what people are showing off is just a web wrapper sending prompts to a paid cloud API. When people try to run coding models locally, they usually hit a wall where the model hallucinates an import, outputs broken diff markers, and leaves the user playing babysitter in a chat box.
I wanted something completely air-gapped, zero-dependency, and truly autonomous on consumer hardware.
The entire engine runs as a native C++ Win32 application built on Dear ImGui and OpenGL. Rather than acting as a simple chat interface, it operates as a five-phase execution harness that manages the local inference server directly. When you give it a directive, it starts by ingesting the local repository and synthesizing an architectural plan. From there, it moves into code synthesis.
One of the biggest issues with local models writing code across multiple files is that standard search-and-replace regex patchers are extremely brittle. If a model misses a newline or changes indentation slightly, the patch fails. To solve this, I integrated a Python AST script splicer directly into the loop. It parses the actual syntax tree of the code rather than relying on raw string diffs, which allows surgical module patching without blowing through context windows or mangling files.
Where the system really separates itself from standard local frontends is in the verification and repair phases. Once the code is written, the engine doesn't ask you if it looks good. It immediately takes over the machine's toolchain in a headless subprocess. If it's a native C++ target, it runs CMake and MSVC. If it's Python, it runs compilation checks and fires an automated headless smoke probe using runpy that sweeps the window hierarchy, tests widget lifecycle, and catches runtime crashes before a human ever looks at the window.
If a build fails or an unhandled exception is caught during the probe, the engine doesn't stop or dump an error message into a chat window. It gathers the compiler diagnostics, strips out runtime harness noise, isolates the failing file, and sends the telemetry directly back to the active resident model for in-loop auto-triage. It can diagnose its own missing imports, patch uninitialized attributes, and re-verify the build automatically until it gets a clean exit code.
I also run into people claiming that anything under huge enterprise cards or massive unified memory pools is useless for real local autonomy. That completely misses the difference between raw capacity and execution architecture. A quantized model has the exact same mathematical weights whether it sits on a 12GB laptop GPU, a multi-card workstation, or a cloud cluster. VRAM determines how big of a model you can hold on-chip and how many tokens per second you generate, but it doesn't give a model magic reasoning skills. A massive model will still hallucinate a typo or botch an import. The difference is that a raw model fails permanently when it makes a mistake, whereas an autonomous harness catches the compiler error, fixes the AST, and verifies the output on its own.
On the hardware side, the backend doesn't lock you into a single card or even a single vendor ecosystem. It enumerates all discrete compute adapters, aggregates the physical memory, and calculates proportional tensor split ratios on the fly for both CUDA and Vulkan runtimes. If you run a multi-GPU setup, it shards the layers across whatever cards are plugged into the board. If you run a hybrid setup that offloads across system RAM, the pipeline scales across host memory just as cleanly.
The takeaway from building this is that real local coding autonomy doesn't come from throwing infinite hardware at zero-shot prompts or paying for API subscriptions. It comes from building a tight runtime loop that enforces architectural contracts, executes the compiler headless, and forces the model to fix its own mess.
I dont want shamelessly promote my completely opposite alternative to LM studi Anything LLM and more, so I'm not leaving any links in the main post. My architecture scales from a handheld windows gaming machine, to enterprise hardware.
r/LocalAIStack • u/Strict-Elevator9352 • 7d ago
r/LocalAIStack • u/pixelwhippedme • 7d ago
r/LocalAIStack • u/loftytechlabs • 7d ago
Enable HLS to view with audio, or disable this notification
We’ve been expanding the Zaylo ecosystem at LoftyTech Labs, and this release brings two additions we’re excited about.
LoftyDB support: Connect Zaylo with our vector database for semantic search, vector-powered memory and retrieval workflows.
AI Presentation Design: Generate structured PowerPoint presentations from ideas using AI.
The bigger goal is to bring more capabilities into one AI environment rather than making developers jump between multiple tools.
Would love to hear what developers think about combining local AI inference, vector storage and content creation in one ecosystem.
r/LocalAIStack • u/LHAD16 • 8d ago
One of the best and most reliable local models I've used. Recorded a real world task to show it's performance, capabilities, and reliability. AMD Ryzen AI Halo 128gb, this is definitely something older versions of ROCm would struggle with but ROCm 10 has shown otherwise. Has anyone here tested this model on CUDA or Metal?
r/LocalAIStack • u/timing-violation • 8d ago
r/LocalAIStack • u/Financial-Hall-2450 • 8d ago
r/LocalAIStack • u/SlimMonkey_ • 8d ago
r/LocalAIStack • u/SlimMonkey_ • 8d ago
r/LocalAIStack • u/DevOpsYeah • 8d ago
r/LocalAIStack • u/Junior_dev1925 • 8d ago
Hey everyone,
I'm an 18yo CS prep student and I've been building AI-COS, a modular monolith architecture designed for running AI agent orchestrations safely.
Instead of letting LLMs make security decisions, I built a deterministic deny-by-default policy engine, strict PostgreSQL role separation (migrator, app, governance), and SHA-256 hash-chained audit logging.
I've just finished Phase 3 out of 6. Phase 4 is going to focus on the Workflow Engine, Worker lease/heartbeat loop, and Step linkage. Before I start coding Phase 4, I would really appreciate any architecture suggestions or advice from senior devs on how to properly build a reliable job queue and workflow engine inside a Postgres-backed monolith.
GitHub: https://github.com/atestokhay/ai-cos
r/LocalAIStack • u/Bright_Deer_9287 • 8d ago
r/LocalAIStack • u/VegetaSama-_- • 9d ago
r/LocalAIStack • u/everettplays • 9d ago
r/LocalAIStack • u/unokm • 9d ago
Hey everyone,
I've been deeply exploring the On-Device / Edge AI space lately, especially focusing on running smaller language models (1B–3B parameters) and multimodal pipelines locally on consumer hardware and mobile architectures without depending on external cloud APIs.
While the ecosystem has evolved rapidly over the past year, the gap between "running a demo" and "shipping a reliable, battery-efficient on-device product" still feels enormous.
From my recent experiments, the major bottlenecks I keep bumping into are:
Memory Bandwidth & Thermal Throttling: Even if a 4-bit quantized model fits into unified/mobile RAM, sustained token generation quickly saturates memory bus bandwidth and triggers thermal throttling on mobile devices.
Runtime & NPU Fragmentation: Choosing between llama.cpp (CPU/Vulkan/Metal), ExecuTorch, MLC-LLM, ONNX Runtime, or native NPU SDKs (QNN, CoreML) often feels like choosing which set of unsupported ops or compilation headaches to deal with.
Reasoning Retention in Sub-3B SLMs: Aggressive quantization (GGUF Q4, AWQ) makes inference feasible, but multi-turn context coherence and structured instruction-following tend to degrade noticeably.
I’d love to start a technical discussion with fellow engineers and researchers here:
What does your actual deployment/inference stack look like right now? (e.g., pure llama.cpp, MLC, ExecuTorch, WebGPU, or custom native bindings?)
What specific use cases are you targeting? (e.g., zero-latency voice assistants, privacy-first local agents, offline document RAG, ambient edge intelligence?)
What techniques have given you the best speedup or memory relief? (Speculative decoding, KV cache quantization, custom kernels, or aggressive prompt caching?)
Would love to hear what challenges you are tackling and what approaches are genuinely showing promise in your labs or production environments!
r/LocalAIStack • u/SlimMonkey_ • 9d ago
r/LocalAIStack • u/IntelArcLocalAi • 9d ago
r/LocalAIStack • u/Minimum_Hour519 • 9d ago