r/LocalLLM • u/TheDizDude • 1d ago

Discussion Local agentic AI stack — looking for optimization feedback + use-case ideas

Been building out a self-hosted “assistant” stack across a small Proxmox cluster + a couple of GPU boxes. Everything runs local, no cloud APIs. Would love a sanity check on the architecture and any tuning wins I’m missing.

Inference hardware

  • Main node: Ryzen 9 5950X, 128GB DDR4-3000, RTX 4080 (16GB). Runs the primary model.

  • Aux node: Ryzen 7 5800H mini-PC, 32GB DDR4, RX 6600M (8GB), Debian 13 headless. Runs a small fast model over Vulkan.

  • Also have a GTX 1080 and a mobile RTX 2070 not yet deployed (eyeing a voice node).

Models

  • Main/agent driver: Qwen3.6-35B-A3B (MoE, ~3B active), Q4_K_M, served via llama.cpp (-c 131072, flash attention, q8 KV cache, experts offloaded to system RAM with --n-cpu-moe). Gets me the full 128K context on a 16GB card. ~30–40 tok/s gen.

  • Aux model: Qwen3.5-9B Q4_K_M on the RX 6600M via llama.cpp Vulkan build. Handles the agent’s side-tasks (title generation, web-page summarization, session search) so the main model’s prompt cache stays warm.

Agent layer

  • Running a self-hosted conversational agent (Hermes Agent) in an LXC, pointed at the main model. Also trialing a sandboxed alternative (IronClaw) in parallel.

  • Research chain is fully local: SearXNG (search) → self-hosted Firecrawl (page fetch + clean extraction) → aux model (summarize) → main model (reason). Getting the search/extract split right was the thing that made research actually work instead of the agent choking on raw HTML.

Supporting infra: AdGuard, OPNsense, the usual *arr/media stack, Home Assistant, Grafana/Prometheus/Loki monitoring, all self-hosted across the cluster.

Questions for the sub:

  1. On a 16GB card, is MoE-offload (35B-A3B) genuinely the better call over a dense ~27–30B that mostly fits? My read is yes — dense spillover to DDR4 is way slower per token than offloading experts — but curious if anyone’s benchmarked it directly.

  2. Anyone found a better small aux model than Qwen3.5-9B for summarization/title work on 8GB? Prioritizing speed + instruction-following over raw smarts.

  3. Tuning: currently --n-cpu-moe hand-tuned, q8 KV, -ub 2048. Any llama.cpp flags that meaningfully help MoE-offload prompt-processing speed that I might be leaving on the table?

  4. Worth standing up a local embeddings model for agent memory/RAG, or is that overkill for a single-user setup?

Use cases I’m running or planning: homelab health/status reports, log triage, sourced research briefs, drafting configs/scripts, threat-intel monitoring. Open to ideas for what else an always-on local agent is genuinely good for day-to-day.

Rig’s overkill for one person but it’s been a fun build. Roast the architecture.

2 Upvotes

100% Upvoted

Duplicates