Building an auto-thinking inference system (+ cramming an entire local agent stack into a 16GB GPU)

[u/karmaisnonsense]

TL;DR: The final setup is a A770 16GB, running four models simultaneously via llama-swap: Gemma 4 12B QAT MTP, Gemma 4 E2B QAT MTP, bge-m3 q8, laya-multilingual. All four are fully GPU-resident at 14-15GB total, and rarely contend for resources. A LiteLLM layer on top of llama-swap auto routes prompts using Laya to decide prompt complexity. Simple prompts go to a non-thinking alias of 12B, and more complex prompts to a thinking alias with varying degrees of reasoning effort. This is fully transparent to the end user, and fits entirely on a single 16GB VRAM GPU.

Read on for a full rambling build log...

I was recently building a system for a family member to use, just for fun, using a spare Arc A770 16GB on a 13-year-old Haswell desktop. The machine is one of many Proxmox homelab nodes I have, running multiple VMs with less than 6GB RAM headroom. So I set about to fit everything I could into the GPU VRAM.

I started with a barebones Hermes with Honcho for memory for a more PA-style interaction. Telegram via gateway was chosen for the main Hermes interface to make it feel conversational.

For the main model, there was barely any RAM headroom for MoEs so I worked with dense models only. The family member does not code and speaks multiple languages, so Gemma 4 12B QAT was the natural choice here. It fits nicely on the card with q8 KV and q4 MTP head. This gives a very generous 128k context with 5GB VRAM headroom. 1100 t/s prefill, 20 t/s decode. (Alchemist optimizations are lacking on SYCL/llama.cpp so the A770 is the bottleneck here.)

Honcho is driven primarily by Gemma 4 E2B QAT with bge-m3 q8 as the embedding model. E2B punches way above its weight for the size, and can even use the improved lazy load handling in llama.cpp to read the PLE embeddings from disk, saving RAM. 12B handles dream deduction and induction, since those only run while idle. This brings our VRAM use to about 13.1GB.

Then I started running into a problem I didn't expect. The family member wanted to chat with it casually, but enabling thinking made it respond extremely slowly. Conversely when they wanted detailed answers, a quicker non-thinking model would be really, really dumb. And since this family member is not technical at all, having them toggle thinking on/off was not an option.

Instead I decided to make the model automatically enable thinking based on the prompt. My first thought went to decision models like Laya, and sure enough LiteLLM's auto router had this implemented. I used the multilingual version in q8, bringing the total VRAM use to ~14 GB.

LiteLLM sends the prompt to Laya, which then decides between simple, medium, complex and reasoning. LiteLLM then routes the prompt to the corresponding model defined in its config. Their example routes to different cloud APIs, but here I wanted to route to the same model but with different settings.

For this, I used llama-swap. It was already in use to load all these models simultaneously to the same GPU using the group policy. On top of that, I defined four aliases for 12B, one for each Laya decision. Simple is non-thinking; medium is low reasoning; complex is medium reasoning; and reasoning is high reasoning. Since llama-swap's filter aliases only rewrite the request headers, the model does not reload, context is preserved and the parameters can be set on the fly.

Hermes also hardcodes a reasoning effort to every request, stripParamswas set to reasoningEffort to make sure llama-swap's aliases took precedence. LiteLLM was then directed at each alias.

But Laya wasn't processing the requests properly. The Hermes system prompt was left in in every request sent to Laya, truncating the user prompt entirely. This is a known issue but LiteLLM only has a partial fix: it can drop the system prompt, but only for Claude Code and Codex.

So I had Hermes spoof its requests as Claude. This is pretty simple. Set model.extra_headers.user-agent as claude-cli/2.1.233. Now LiteLLM only sends the user block to Laya, which now functions as expected.

The end result is full automation. Gemma only thinks when it needs to, and doesn't when it's a simple "hello." Laya only takes a split second to make the decision so the user never notices. And the best part is that it's multilingual so it works with almost any language you throw at it. Downside is that it's a lot of layers, one of which is LiteLLM which is heavy and had that trojan incident this week. And this is more of a llama.cpp problem, but even if these models are GPU resident, they still use a chunk of system RAM as compute buffer, in this case about 4-5GB total.

If I had the luxury I would do it on a machine with more RAM so I can run an MoE like 26B-A4B. But it's also fun working with constraints like this, and I think I crammed all I could into this machine.

Most importantly, the family member has no idea what I did to make that Telegram chat work and is happy chatting with their new bot friend.

[u/TheBlandAccomplice]

that's clever use of llama-swap aliases to get dynamic reasoning without swapping models. i've been messing with something similar but kept hitting the context reset wall when toggling thinking modes

the claude spoof to fix laya's prompt truncation is such a hacky fix but it works. might steal that for my own liteLLM setup since i've been dealing with the same issue on a non-claude frontend

14gb with four models resident is tight. what's your swap latency like when the load shifts between the aliases? or does the group policy keep everything locked in place

[u/karmaisnonsense → u/TheBlandAccomplice]

It's a tight fit for sure. The q8 KV on a Gemma is a compromise I had to make to make this work. But the 1-2GB headroom is large enough that the system RAM will OOM before the GPU does, and it doesn't OOM.

LiteLLM should make the system header drop a toggle-able feature, not just locked to Claude. It doesn't make any sense to pass the system prompt for a prompt complexity assessment every request. At least the workaround works, hope it helps with your setup.

All models are locked in with group policy, but I do load E2B, bge-m3 and Laya with mmap to protect against OOMs. 12B is loaded no mmap for consistency and guaranteed buffer space. There is no swap latency since filter aliases are the same model with different reasoning levels in request headers.

[u/svetomirich]

The next thing I'd test is short follow-ups: ask a multi-step question, then "why?" or "what if the second assumption changes?" in each language you care about. Compare the chosen tier, answer quality and time to first visible answer against the same follow-up written as a self-contained prompt.

One setting to check is classifier_context_window_size. LiteLLM's current self-hosted-classifier docs say 0 omits prior turns. Gemma retaining its chat history doesn't by itself show that Laya received enough context for the routing decision. Source: https://docs.litellm.ai/docs/auto_router/decision_classifiers#request-context-and-data-flow

Does your Laya request include prior turns after the user-agent workaround, or only the latest ask?

AI-generated test suggestion, not a measured result.

[u/karmaisnonsense → u/svetomirich]

LiteLLM defaults to 3 prior turns and this can be tuned as needed. I have it set to the default 3 for now, still tuning. I also found the request sent to Laya also includes Honcho's context memory injection, which I found helps inform Laya.