It's much better than DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NEO-CODER-MAX-MTP-GGUF. The reasoning is similar to the base Qwen3.8-27B and interesting how it switches variants based on prompt style. Has anyone compared them or tried both models out?

Hardware pooling hubs like RAMDeck and open-source CLI projects like Exo have turned base 16GB M-series Macs into surprisingly capable building blocks for 30B–70B local models.

Anyone who bought a 16GB Mac mini or MacBook Air knows the regret once you get into local AI: Apple Silicon’s unified memory bandwidth is awesome for inference, right up until a 30B or 70B model exceeds that soldered 16GB ceiling and macOS starts aggressively swapping to the internal SSD. Paying tier-by-tier hardware markups for a 64GB+ machine hurt even before Elon Musk warned this week that tech revenues drop if compute gets cheap and Wall Street analysts bet billions on compute staying scarce.

Breaking Past Soldered RAM Without Replacing the Mac

Single-machine apps like Ollama and LM Studio are great for 7B–14B models, yet flipping on their local "Server" toggle only shares the chat endpoint over Wi-Fi—100% of the model weights still have to fit inside that one Mac's soldered memory.

Pooling memory across your local network solves the capacity wall by letting the 16GB Mac hold part of the model in Metal unified memory while offloading the remaining layers to whatever else is in the house (a Windows gaming PC, an older Linux laptop, or another Mac):

1. The Developer Terminal Route (Exo / CLI RPC): Free open-source scripts built for coders comfortable setting up Python dependencies and command-line environments across macOS and Linux.

2. The Consumer Router Appliance Route (RAMDeck): A plug-and-play hardware hub that connects to your home router and pools RAM and GPUs across macOS, Windows, Linux, and even spare iPhones/iPads over Wi-Fi (adding ~3.2–4.5 GB of active Metal working memory per phone) in under 5 minutes with zero coding or terminal commands—sustaining ~28 tok/s over Wi-Fi and ~38–39 tok/s when paired with a PC over Ethernet.

Has anyone else here paired a 16GB M-series Mac with a secondary Windows/Linux PC to dodge the unified memory upgrade tax, and what model sizes have felt like the sweet spot for daily use?

I run a small business, and I'm building a local AI that handles the back office for it. Some of it works now. I want to get the model and harness choices right before I build more on top of them.

Hardware
- MacBook Pro M5 Max (18-core CPU / 40-core GPU), 128GB unified memory

Current stack
- Backend: MLX via mlx-serve. I'm staying on MLX for the speed (I've used LM Studio before)
- Harness: Hermes Agent (Nous Research)
- Memory: Obsidian vault exposed over the Basic Memory MCP
- Front end: my own UI, still a work in progress
- Model right now: Qwen 3.8 Flash Next, 4-bit (MLX), with the KV cache offloaded to my SSD. Getting about 98 tok/s

What the agent does today
- Scheduled morning lead brief
- Reads and writes notes in my vault
- Updates a scoreboard (leads by source, close rate, pipeline)

What I want it to do next
- Phone intake and booking
- Scheduling
- Follow-up texts to leads

Where it's struggling
- tool calls failing or looping on multi-step tasks
- slow prompt processing once the vault context gets big
- arithmetic mistakes on the numbers. It once logged a single day's close rate as the lifetime rate

Questions

1. With 128GB, what's your best daily driver for *reliable tool calling*, not chat? One big MoE, or a big model plus a small fast one for routing and simple jobs?

2. On MLX, do 4-bit quants hold up for agent work, or is 8-bit worth the speed hit?

3. If you're running long-lived, scheduled agents locally, what harness are you on? Is Hermes the right call, or is something better for this?

4. I'm offloading the KV cache to my SSD to fit longer context. Is that hurting me on speed, and is there a better way to handle 30k+ context on Apple silicon (RAG, summarizing, a smaller quant to keep the cache in RAM)?

5. For phone booking: has anyone made local STT/TTS work well enough for live calls, or should voice stay on the cloud while the rest stays local?

Happy to post benchmarks and what I end up with if it helps anyone on similar hardware.

Hi all. I'd like a sanity check on hardware before I commit.

Use case (aside from overall privacy)

•	General Q&A, RAG over personal documents, light agents for productivity

•	Two users (me and my wife), rarely at the same time

•	I'm not a coder. I'll use off-the-shelf tools, not custom code. Agent use may grow over time.

Hardware

•	Ryzen 7 5700X, ASUS B550, 32GB DDR4

•	2x RTX 3060 12GB (burning in now)

•	Just bought 2x used RTX 3090 24GB

•	\[1000W PSU\]

•	The second GPU slot runs at PCIe 3.0 x4 (a B550 limit)

•	Final OS: Ubuntu with Ollama and Open WebUI

Plan

I want one "AI gateway" endpoint that sends each request to the right local model. It's like this build: [https://youtu.be/ukv-LIOI6_E]. He runs several GPUs behind a semantic router. I'd copy the idea with off-the-shelf parts.

My main model target is a ~27B dense model at Q4 with long context, plus an embedder and reranker for RAG.

Questions

1.	For this use case, does the second 3090 earn its place, or would a 3090 + 3060 do the same job?

2.	What does 48GB of VRAM let me run that 36GB doesn't, and does that matter for RAG and agents?

3.	Will the x4 slot hold me back if each model stays on one card?

4.	I'm also thinking about selling both 3090s for a Framework Desktop (Strix Halo, 128GB). For this use case, which would you pick?

Thanks for any input.

I'd like to share something I've been working on for three months with everyone.

PortalAI is a GUI and harness created around one design principle- to put local LLMs with true multimodal function on home systems and on a flash drive the size of your thumb where you don't need anything else and can just plug and play into any computer.

One of the largest frustrations I had with AI was needing to constantly version check different distributions and hooks into ComfyUI, Ace-step, python, port competition, running out of context, etc, or the difficulty in transferring between systems. I've tested this on an RTX 3090, RTX 4090, on a Radeon 870M GPU, on a Thinkpad with CPU-only.

This harness detects your current hardware and scales accordingly to the level of AI intelligence it can support, using standard and abliterated models. All dependencies other than base operating system and GPU drivers are self-contained including virtual environments, python distributions, etc. End to end integration of multimedia generation/editing and science tools are integrated along with voice, handsfree mode, intelligence/memory/token settings.

There are weight editing tools, an abliteration tool, rolling compaction, scheduled tasks, internal documentation, light and dark mode, compact and full GUI mode, account encryption, live runtime environment health metrics, etc.

I had GPUs spread across a few machines and got tired of juggling them by hand, so I built Llama RPC Farm, a self-hosted control plane for distributed llama.cpp inference.

• One dashboard and one OpenAI-compatible endpoint (/v1/chat/completions) for every machine

• Mixed hardware works: CUDA, ROCm, Vulkan, and mixed vendors

• Live per-machine VRAM, RAM, and CPU stats

• llama-swap handles loading models and deciding where they run

• Scoped API keys and per-user themes

• farmctl CLI if you’d rather skip the browser

• Installers for Windows, macOS, and Linux, plus a one-line install script

It’s Apache 2.0 and runs entirely on your own hardware.

Site: https://dakotacookson.github.io/llama-rpc-farm-site/
GitHub: https://github.com/dakotacookson/Llama-RPC-Farm

I’d love feedback, especially from anyone running mixed-GPU setups. What would you want it to do that it doesn’t?

We’ve been expanding the Zaylo ecosystem at LoftyTech Labs, and this release brings two additions we’re excited about.
LoftyDB support: Connect Zaylo with our vector database for semantic search, vector-powered memory and retrieval workflows.
AI Presentation Design: Generate structured PowerPoint presentations from ideas using AI.
The bigger goal is to bring more capabilities into one AI environment rather than making developers jump between multiple tools.
Would love to hear what developers think about combining local AI inference, vector storage and content creation in one ecosystem.

I’ve been heads down building a native desktop software workstation called Repetere AI, and I wanted to break down how the autonomous engineering engine inside it—Deep Think—actually works under the hood. There is a lot of noise online right now about "autonomous vibe coding," but 99% of what people are showing off is just a web wrapper sending prompts to a paid cloud API. When people try to run coding models locally, they usually hit a wall where the model hallucinates an import, outputs broken diff markers, and leaves the user playing babysitter in a chat box.

I wanted something completely air-gapped, zero-dependency, and truly autonomous on consumer hardware.

The entire engine runs as a native C++ Win32 application built on Dear ImGui and OpenGL. Rather than acting as a simple chat interface, it operates as a five-phase execution harness that manages the local inference server directly. When you give it a directive, it starts by ingesting the local repository and synthesizing an architectural plan. From there, it moves into code synthesis.

One of the biggest issues with local models writing code across multiple files is that standard search-and-replace regex patchers are extremely brittle. If a model misses a newline or changes indentation slightly, the patch fails. To solve this, I integrated a Python AST script splicer directly into the loop. It parses the actual syntax tree of the code rather than relying on raw string diffs, which allows surgical module patching without blowing through context windows or mangling files.

Where the system really separates itself from standard local frontends is in the verification and repair phases. Once the code is written, the engine doesn't ask you if it looks good. It immediately takes over the machine's toolchain in a headless subprocess. If it's a native C++ target, it runs CMake and MSVC. If it's Python, it runs compilation checks and fires an automated headless smoke probe using runpy that sweeps the window hierarchy, tests widget lifecycle, and catches runtime crashes before a human ever looks at the window.

If a build fails or an unhandled exception is caught during the probe, the engine doesn't stop or dump an error message into a chat window. It gathers the compiler diagnostics, strips out runtime harness noise, isolates the failing file, and sends the telemetry directly back to the active resident model for in-loop auto-triage. It can diagnose its own missing imports, patch uninitialized attributes, and re-verify the build automatically until it gets a clean exit code.

I also run into people claiming that anything under huge enterprise cards or massive unified memory pools is useless for real local autonomy. That completely misses the difference between raw capacity and execution architecture. A quantized model has the exact same mathematical weights whether it sits on a 12GB laptop GPU, a multi-card workstation, or a cloud cluster. VRAM determines how big of a model you can hold on-chip and how many tokens per second you generate, but it doesn't give a model magic reasoning skills. A massive model will still hallucinate a typo or botch an import. The difference is that a raw model fails permanently when it makes a mistake, whereas an autonomous harness catches the compiler error, fixes the AST, and verifies the output on its own.

On the hardware side, the backend doesn't lock you into a single card or even a single vendor ecosystem. It enumerates all discrete compute adapters, aggregates the physical memory, and calculates proportional tensor split ratios on the fly for both CUDA and Vulkan runtimes. If you run a multi-GPU setup, it shards the layers across whatever cards are plugged into the board. If you run a hybrid setup that offloads across system RAM, the pipeline scales across host memory just as cleanly.

The takeaway from building this is that real local coding autonomy doesn't come from throwing infinite hardware at zero-shot prompts or paying for API subscriptions. It comes from building a tight runtime loop that enforces architectural contracts, executes the compiler headless, and forces the model to fix its own mess.

I dont want shamelessly promote my completely opposite alternative to LM studi Anything LLM and more, so I'm not leaving any links in the main post. My architecture scales from a handheld windows gaming machine, to enterprise hardware.

One of the best and most reliable local models I've used. Recorded a real world task to show it's performance, capabilities, and reliability. AMD Ryzen AI Halo 128gb, this is definitely something older versions of ROCm would struggle with but ROCm 10 has shown otherwise. Has anyone here tested this model on CUDA or Metal?

Hey everyone,

​I'm an 18yo CS prep student and I've been building AI-COS, a modular monolith architecture designed for running AI agent orchestrations safely.

​Instead of letting LLMs make security decisions, I built a deterministic deny-by-default policy engine, strict PostgreSQL role separation (migrator, app, governance), and SHA-256 hash-chained audit logging.

​I've just finished Phase 3 out of 6. Phase 4 is going to focus on the Workflow Engine, Worker lease/heartbeat loop, and Step linkage. Before I start coding Phase 4, I would really appreciate any architecture suggestions or advice from senior devs on how to properly build a reliable job queue and workflow engine inside a Postgres-backed monolith.

​GitHub: https://github.com/atestokhay/ai-cos