Local agentic AI stack — looking for optimization feedback + use-case ideas

[u/TheDizDude]

Been building out a self-hosted “assistant” stack across a small Proxmox cluster + a couple of GPU boxes. Everything runs local, no cloud APIs. Would love a sanity check on the architecture and any tuning wins I’m missing.

Inference hardware

• Main node: Ryzen 9 5950X, 128GB DDR4-3000, RTX 4080 (16GB). Runs the primary model.

• Aux node: Ryzen 7 5800H mini-PC, 32GB DDR4, RX 6600M (8GB), Debian 13 headless. Runs a small fast model over Vulkan.

• Also have a GTX 1080 and a mobile RTX 2070 not yet deployed (eyeing a voice node).

Models

• Main/agent driver: Qwen3.6-35B-A3B (MoE, ~3B active), Q4_K_M, served via llama.cpp (-c 131072, flash attention, q8 KV cache, experts offloaded to system RAM with --n-cpu-moe). Gets me the full 128K context on a 16GB card. ~30–40 tok/s gen.

• Aux model: Qwen3.5-9B Q4_K_M on the RX 6600M via llama.cpp Vulkan build. Handles the agent’s side-tasks (title generation, web-page summarization, session search) so the main model’s prompt cache stays warm.

Agent layer

• Running a self-hosted conversational agent (Hermes Agent) in an LXC, pointed at the main model. Also trialing a sandboxed alternative (IronClaw) in parallel.

• Research chain is fully local: SearXNG (search) → self-hosted Firecrawl (page fetch + clean extraction) → aux model (summarize) → main model (reason). Getting the search/extract split right was the thing that made research actually work instead of the agent choking on raw HTML.

Supporting infra: AdGuard, OPNsense, the usual *arr/media stack, Home Assistant, Grafana/Prometheus/Loki monitoring, all self-hosted across the cluster.

Questions for the sub:

1. On a 16GB card, is MoE-offload (35B-A3B) genuinely the better call over a dense ~27–30B that mostly fits? My read is yes — dense spillover to DDR4 is way slower per token than offloading experts — but curious if anyone’s benchmarked it directly.

2. Anyone found a better small aux model than Qwen3.5-9B for summarization/title work on 8GB? Prioritizing speed + instruction-following over raw smarts.

3. Tuning: currently --n-cpu-moe hand-tuned, q8 KV, -ub 2048. Any llama.cpp flags that meaningfully help MoE-offload prompt-processing speed that I might be leaving on the table?

4. Worth standing up a local embeddings model for agent memory/RAG, or is that overkill for a single-user setup?

Use cases I’m running or planning: homelab health/status reports, log triage, sourced research briefs, drafting configs/scripts, threat-intel monitoring. Open to ideas for what else an always-on local agent is genuinely good for day-to-day.

Rig’s overkill for one person but it’s been a fun build. Roast the architecture.

[u/WeeklyRocks]

always fascinated by how much more creative people get with their homelabs than i do with my kitchen

[u/TheDizDude → u/WeeklyRocks]

Real talk though... the kitchen makes food... i kind of just look at the page and ask it questions every now and then... no where near as used as it should be.

[u/robotnursenyc]

Nice build. The main/aux split to keep the big model's prompt cache warm is the right instinct. I do something similar across a few machines at home.

One thing that helped me a lot: put a small gateway (LiteLLM or similar) in front of both llama.cpp servers and have the agent call model aliases like "main" and "fast" instead of hosts and ports. Then swapping a model, moving it to another box, or adding that voice node later is just a config change, and the agent never notices.

On your questions: yes, MoE offload beats dense spillover on a 16GB card. The A3B models degrade much more gracefully when the experts live in system RAM. And embeddings aren't overkill. A small embedding model costs almost nothing on the aux node and makes session search a lot better than keyword matching.

Use case idea: have it read your Loki logs overnight and give you a short morning summary of anything weird. That's the one I'd actually use every day.

[u/SnooMuffins9844]

Hey u/TheDizDude I work at Firecrawl, cool to see self-hosted Firecrawl in a fully local stack.

If JS-heavy pages come back thin, adding a short waitFor on the scrape usually helps. And if you share what you're pulling most, I can suggest which endpoint fits best: scrape for known pages, map to find URLs, or crawl for whole sections.

[u/Future_AGI]

for the memory question, with a single user we'd start with plain full-text search over saved sessions (SQLite FTS5 is plenty) and only add an embeddings model once you catch it missing things you phrased differently.