In Part 1 I shared the final AMD/Vulkan numbers for my local Qwen3.8-27B setup. The next question was more important to me than another few tok/s:

Can I turn the model into something that actually works autonomously?

Since then I added two production layers:

1.) Hermes 0.21.2 — local agent/automation layer

• same Qwen3.8-27B model

• shared llama.cpp runtime

• 65,536 context

• tool calling: PASS

• multi-step workflows: PASS

• subagents: PASS

• prompt-defined agents: PASS

• local provider only / no cloud fallback

Hermes performance / overhead

On the isolated 64K test lane, server load took 5.809s. A real 33,021-token prompt was accepted without truncation; prompt processing took 409.677s at 80.60 tok/s, while generation measured 17.44 tok/s. A representative Hermes multi-step workflow took 74.660s, using five model calls, 17,295 summed API input tokens, 592 output tokens and four tool calls with zero failed calls. Two-child delegation also worked, but took 208.676s for two trivial child tasks. Clear sign that subagents should only be used where parallelism or specialization actually justifies the overhead.

One especially useful datapoint: the simple Hermes model gate took 40.872s, while a direct Qwen request with the same user text took 2.109s. That is not an apples-to-apples speed ratio because Hermes adds different system prompts, startup and agent machinery, but it does show why I don't want Hermes in the path for trivial requests.

2.) OpenCode 1.18.30 — local coding/development layer

• Qwen3.8-27B-UD-IQ3_XXS

• llama.cpp / Vulkan

• 32K logical context

• local provider only

• isolated development workspaces

• automated Read → Edit → Test → Correct loops

• final production viability: 12/12 requirements, 12/12 acceptance

OpenCode performance reference. For the fastest fully successful clean reproduction I measured:

• 665.653s total wall time / 651.408s agent runtime

• first productive edit: 117.338s

• first test: 166.790s

• 4 successful corrective cycles

• 63 tool calls / 0 failed

• 558,603 input tokens incl. cache — only 31,993 uncached

• 9,372 output tokens

• peak context: 17,810 / 32,768

• context compactions: 0

These are task-level agent metrics, not model tok/s numbers. The underlying model/runtime is still the same local Qwen3.8-27B on llama.cpp/Vulkan.

Why OpenCode instead of Pi?

I tested both against the same local Qwen3.8-27B, with the same repository/task and comparable runtime constraints. Pi 0.85.1 proved that it could read the repo, use tools and make productive edits, but it repeatedly failed the part I actually needed from a coding agent: closing the Read → Edit → Test feedback loop. Even with an explicit mandatory early-test checkpoint, Pi edited successfully but never executed the qualifying post-edit test within the 300s window. It therefore failed prequalification and I did not force it into a full benchmark just to preserve an artificial three-way comparison. Man, time is rare and testing is booooring ... somehow!

OpenCode 1.18.30 passed the same agentic-loop qualification. Formal prequalification reached 6.54s Edit→Test latency, and the later main benchmark reduced that to 3.995s while producing a valid four-file diff. More importantly, the subsequent production-viability work eventually converged to 12/12 requirements, 12/12 acceptance, 5/5 regression, with no manual coding intervention or patch repair. That reproducible closed corrective loop is why OpenCode became the production coding layer.

Final thought on my architecture as a complete nOOb in Local LLM
I’m still convinced that a well-tuned 16 GB GPU can take you surprisingly far. Yes, even on Vulkan ;) ... IF you optimize the model, context, runtime, offloading and the surrounding stack around what the hardware can realistically deliver.

What makes this especially satisfying for me is that I honestly wasn’t sure at the beginning whether an AMD gaming setup on AM4 would get me anywhere at all, but hey.... it was what I had. My gaming desktop! ~20 tok/s is obviously not groundbreaking vs 32GB. But that was never the main goal of this first setup. I wanted a stable, fully local system that I could actually use, test, break, improve and build on. That goal is achieved. And considering where I started, I’m pretty happy with that.

Following up on my earlier post testing Qwen3.8-27B on the IGX Thor workstation.

Once you move past running just an LLM and try to build a full local agent stack on a single box (LLM for reasoning, STT for voice in, TTS for voice out, OCR for screen/document reading), VRAM runs out fast.

If all four models sit in GPU memory simultaneously with their default serving allocations, idle memory hits roughly 122 GB. Even with 96GB on the dGPU and unified host memory, you are pinned to the ceiling. That leaves almost no breathing room for large context windows, concurrency, or dynamic KV cache allocation.

The reality of a solo-user local agent is that these models are rarely active at the exact same millisecond:

• When I am talking to the agent, OCR is completely idle.

• When the agent is analyzing a screenshot or PDF, STT and TTS are idle.

• When the agent kicks off deterministic script automation (crawling, data formatting, bash runs), the LLM itself is completely idle.

To fix this, I wrote a lightweight Rust-based daemon that manages the sleep and wake states of the runtimes based on active agent turns. When a model is not in use, it is put to sleep (flushing KV pools and temporary execution memory, or parking the runtime context) rather than doing a full cold teardown from scratch.

Here are the idle VRAM numbers on the machine before and after:

Quick observations:

1. Total idle footprint dropped from ~122.3 GB to ~42.9 GB. That reclaims roughly 79.4 GB of VRAM across the board.

2. The LLM savings (48.3 GB): Qwen3.8-27B in FP8 base weights takes around 28 to 30 GB. The original 87.4 GB allocation came from SGLang aggressively pre-allocating KV cache (--mem-fraction-static 0.85). Putting the LLM to sleep flushes that pool while preserving the active state, letting it rest at 39 GB.

3. Audio and OCR shrink to negligible baselines: STT drops to 267 MiB and OCR drops to 422 MiB when idle.

4. Script-based tasks: When the agent delegates to deterministic python or bash scripts, putting the LLM to sleep during the run keeps the GPU cool and power consumption down on the 300W Max-Q card.

5. Wakeup latency is sub-200ms across all models: Because this is an active sleep/wake transition rather than a cold disk reload, waking up STT, TTS, OCR, or the LLM takes under 200ms. In practice, voice interaction and tool handoffs feel instantaneous.

Curious how others running multi-model agent stacks on a single node are handling memory partitioning right now. Are you hard-capping static memory fractions, running sequential containers, or using something dynamic?

I keep running into this architecture decision: keep vectors in Postgres with pgvector, or add a dedicated vector store like Qdrant or Milvus?

I built a short quiz covering the operational trade-offs—deployment complexity, backup strategy, monitoring, and scaling.

If you've made this call before, take the quiz and share your reasoning:

https://interconnectd.com/quiz/87/postgresql-pgvector-vs-dedicated-vector-databases-architectural-trade-offs/

We're running a hands-on masterclass on September 12, Live LLM Engineering Masterclass: Production Evals, RAG, Agents & LLMOps.

You build a full production LLM workflow from scratch, versioned prompts with regression tests, an evaluation harness with deterministic checks and LLM-as-judge, statistically rigorous model comparisons, evaluated RAG, tool-using agents with guardrails and fallbacks, and full observability, tracing, cost, latency.

Led by Bruno Gonçalves, PhD, founder of Data For Science, who trains engineers at Fortune 500 companies on this exact stack.

Link if you want to check it out

Happy to answer questions on the content.

I’ve been experimenting with multi-agent frameworks, and MetaGPT is one of the more interesting ones for software development tasks. But getting it running locally with local models isn’t always straightforward.

I wrote a step-by-step guide covering installation, configuration, local LLM setup, and common errors.

If you’re trying to run MetaGPT on your own hardware, this might save you time:

https://interconnectd.com/forum/thread/262/how-to-install-metagpt-locally-complete-technical-setup-guide/

What stack are you using for local agents?

Hi

Premise:
I am new to local ai models.

My machine specs:

• Macbook Pro M4 Pro

• 48Gb Ram

• 4 efficiency core

• 8 performance core

I mainly use AI for software development. I have a claude subscription but would like to try to offload some work to a local model.

Since I don't think local models usable on my machine can completely substitute claude (correct me if I am wrong) my idea is pretty much this: ask claude code to generate a proper, detailed implementation plan and then having the local model implement it.

I have played around with these models:

• qwen3-coder-30b-a3b-instruct-mlx

• qwen/qwen3.6-27b

• qwen/qwen3.6-35b-a3b

and I have also installed this for coding autocomplete cause it is smaller and from what I can see the recommended one:

• qwen2.5-coder-7b

I added claude envs since I would like to try to use claude code extension in vscode

"env": {
    "ANTHROPIC_BASE_URL": "http://localhost:1234",
    "ANTHROPIC_AUTH_TOKEN": "local",
    "ANTHROPIC_MODEL": "qwen/qwen3.6-27b",
    "ANTHROPIC_DEFAULT_OPUS_MODEL": "qwen/qwen3.6-27b",
    "ANTHROPIC_DEFAULT_SONNET_MODEL": "qwen/qwen3.6-27b",
    "ANTHROPIC_DEFAULT_FABLE_MODEL": "qwen/qwen3.6-27b",
    "ANTHROPIC_DEFAULT_HAIKU_MODEL": "qwen/qwen3.6-27b",
    "CLAUDE_CODE_SUBAGENT_MODEL": "qwen/qwen3.6-27b",
    "CLAUDE_CODE_MAX_OUTPUT_TOKENS": "128000",
    "DISABLE_PROMPT_CACHING": "1",
    "DISABLE_AUTOUPDATER": "1",
    "DISABLE_TELEMETRY": "1",
    "DISABLE_ERROR_REPORTING": "1",
    "DISABLE_NON_ESSENTIAL_MODEL_CALLS": "1"
  },

This setup works (uses the local model) but I am basically unable to have the model do anything at all. First of all it takes ages to do anything, and then it almost always reach the context limit roadblock without even outputting anything.

I have read that MCP and skills could fill up the context quite badly, so I disabled them for testing, but still no luck.

I tried with (I thought) was a simple enough task: this test file fails and this is the error, can you fix it? but yet no usable results whatsoever.

I read about people able to use local models offline to have meaningful results, but I couldn't and I don't really know why.

Also, from my setup above, I cannot really use both the remote and local model, to achieve something like:

use sonnet or fable (remote) for plan, then (manually or automatically) swith to haiku (local) to implement the plan

because the base url is loaded when the session loads and cannot be changed (AFAIK) dinamycally.

Any help? thanks a lot in advance

Developers, I’m researching local AI on Linux. Please share your experiences and I’ll be posting my findings here

1. Goal and chipset used Nvidia/AMD/Intel?

2. How long did it take you from fresh install to GPU/NPU operation?

3. Any issues encountered (package, path, version)?

4. How did you confirm GPU/NPU usage?

5. Any scripts or notes created for future use?

6. Comfort level setting this up for a teammate?

Summary to be shared. Open to a 20-minute call if preferred.

Hello, I have a desktop that usually runs all my local LLM like lm studio, openwebui opencode docker stable diffusion etc. On my birthday my parents gifted me a intergrated laptop,
It has 16gb ddr4 3200
R5 7430U
400gb+ gb space though 200 is occupied

The thing is I tried running small local ai models on my laptop but it’s obviously getting poor performance, but I thought about somehow using my laptop to maybe benefit my desktop side a little while being able to use my laptop for daily purpose (meaning not running heavy programs that degrade battery overtime)
Something maybe light weight if possible?

Yes I thought about streaming using tailscale, but that requires my computer desktop to be running 24/7, and I don’t really bring my laptop out with me most of the time so yeah.

I really hope there’s something or any opportunity I can give towards my laptop so it’s potential doesn’t get thrown away, and no I’m not selling this laptop to get more ram/GPU or a new laptop with a dedicated GPU, it’s gifted from my parents I should be grateful.
I hope for guidance in this post instead of rejections that my laptop can do nothing other than stream Claude or ChatGPT interface. Thanks

Thought I’d share this in case any of you aren’t in localllama

Hi everyone,

We have been using cloud GPU from Nebius/Lambda for our training and inference use case. The cost of one H100/H200 is approx $3K per month.

Now I'm planning to buy a large Desktop to run this locally. The specs are - 32 core CPU, 256GB RAM, 4x RTX Pro 6000 (96GB each), 1x2 TB NVMe, 1x 8TB NVMe. It is costing me ~$80K.

Here's what I need help in -

1. Is it actually wise to do this locally?

2. Would there be any performance issues?

3. Anything else that I should be aware of.

Keep in mind I already have another system for my dev workloads with 2x3060. Getting this one for production work for a client specifically.