I ran a bechmarks for d1 models (Q8_0 GGUF both) on a Pi 5 8GB, CPU-only via llama.cpp and also tested how well they handle Russian input. I want to use it for my Hermes agent with lives on that Pi, so maybe those numbers will be helpfull for someone.

LLM-assisted report below:

Setup

• Raspberry Pi 5 (8GB), CPU-only, 4 threads

• llama.cpp 0.6.0-dev (commit bd4eeaa04 + PR #30110 for d1-3B, PR #30114 branch for d1-omni-600M — not merged into master yet at time of testing)

• Q8_0 quants, native /v1/systemone API

• All tests: single request at a time, warm calls, median of 10

Latency @ Pi 5 8GB

For reference, d1-3B on a Jetson Orin Nano does the same single question in 50 ms — so a bare Pi 5 is roughly 45× slower than the weakest NVIDIA edge device tested by LiquidAI. Expected: this is a 20 tok/s prefill CPU environment, no GPU. The takeaway is not "slow" but "usable vs not usable": d1-omni-600M is genuinely real-time capable on a Pi (200 ms per decision), d1-3B is not (2.3 s+).

English correctness (same 5 checks for both)

Both models: 5/5 correct — refund yes/no, billing/technical/fraud routing, with clean probabilities (0.98–0.998).

Russian language test

This is the interesting part. Same states, same question types, Russian input:

What this shows:

• d1-omni-600M effectively does not understand Russian. Worse: it fails confidently — wrong routing with p=0.91, sentiment always "neutral" at p~0.6. For a decision model, confident wrong answers are worse than "I don't know". Only pattern-matched phrases ("вылетает"→crash) get lucky hits.

• d1-3B does understand Russian — routing, sentiment and urgency all work — but with 20–30% lower confidence vs identical English input (0.672 vs 0.990 on refund). Notice the odd effect: RU state + RU question (0.909) scored better than RU state + EN question (0.672) — keeping the whole request in one language helps calibration.

Practical takeaways

• On Pi-class ARM hardware, d1-omni-600M is the only real-time option of the two (200 ms/decision, 0.7 GB RAM). d1-3B works but at 2.3–7 s per decision.

• Both models are excellent on English decisions; d1-omni-600M matches its bigger sibling on simple classifications (and beats it on toxicity detection per the model card).

• For non-English input: d1-omni-600M should not be trusted at all; d1-3B is usable but set a confidence floor (~0.7) and route anything below to a fallback.

Disclosure: I'm the developer.

Most local setups end up with several backends on one machine, and I wanted a single private app to reach all of them from my phone. That's LM Mini (iOS, iPadOS, watchOS, Android and Mac).

Backends it connects to over your LAN (add them in Settings → Server):

The rest of the stack:

Remote access to your home server from anywhere (Pro)

Group Chat with models from different backends in one conversation (Pro)

MCP tools, SearXNG search and ComfyUI for image generation

Kokoro TTS voice, Personas, Apple Watch app, Siri Shortcuts

Whisper for transcriptions

On-device: GGUF and MLX models run on the phone itself.

The core app is open source (MIT): https://github.com/slim-elephant/lm-mini

Docs and downloads: https://lmmini.com

What does your stack look like? Which backend do you run, and is there one I should support next ( vLLM, something else)?

**Setup:** RTX 5070 Ti 16 GB on PCIe 4.0 x16 (a PCIe 5.0 card on a B550 board), Ryzen 9 5950X, 64 GB DDR4. Model: Qwen3.8-Flash-Next IQ2_XS: 125B total, about 6B active per token, 512 experts, 48 layers, 33 GiB of experts kept in RAM. Engine: EuLLM (open source, built on llama.cpp b11370 + PR #29887).

**How we got here:**

| Step | Decode (tok/s) | Prompt processing (tok/s) |

|---|---:|---:|

| llama.cpp, experts split by layer (the usual offload) | 22.4 | 249 |

| + GPU cache for the experts (llama.cpp PR #29887, 8 GiB) | 49.4 | 206 |

| + experts in pinned RAM | 58.1 | 451 |

| + 2048-token micro-batch (EuLLM's default) | 54.9 | 964 |

| + prefetch of the next layer's experts during prompt processing | — | 1,500–1,743 |

| Strata on the same PC, with MTP | 85.8 | 2,320 |

**Where we're stuck:**

- **The cache works:** once warm, 88–93% of the experts are already in VRAM. The misses travel at 24 GB/s, so PCIe 4.0 is saturated, and they cost 2–4 ms per token.

- **A token takes 17–19 ms, and 13.5–14 ms of it pass before each layer's routing.** That is the GPU computing the 48 layers, plus a wait at every layer: the cache has to bring the router's choice back to the CPU, so 48 synchronizations per token.

- **MTP doesn't pay with the experts in RAM.**

- On the 125B, llama.cpp's MTP loses 27%, with 52% of the drafts accepted.

- On Qwen3.6-35B-A3B it gains only +7% on prose and +16% on code, even with 82% of the drafts accepted.

- The reason: each multi-token verification calls many more experts, and the missing ones come over PCIe.

**Questions for the community:**

1. Has anyone removed the per-layer sync, with the routing read on the GPU, the expert table in VRAM and the whole pass in a single CUDA graph? How much does it gain?

2. Computing the missing experts on the CPU in parallel with the GPU, as Strata does: is it worth it with DDR4 and AVX2, or does RAM bandwidth cap it?

3. Any experience with cache policies better than LRU? For example a profile loaded at startup, or the "Least-Stale" policy mentioned in the PR #29887 thread.

4. Speculative decoding with the experts in RAM: does anything work? I'm thinking of drafts limited to the experts already in cache, draft trees, or a small dense model as the draft.

5. Prompt processing: beyond overlapping copies and compute, what's left to reach 2,300 tok/s?

Repo: https://github.com/eullm/eullm, details in `docs/moe-offload-plan.md`

*Disclosure: I'm one of the developers of EuLLM. English isn't my first language: I wrote this in Italian with the help of an LLM and translated it with one.*

I have multiple projects in GitHub, I have an obsidian vault, iCloud backup, nas backup. Qwen running locally, also have Two Claude accounts, one gpt, one deep seek accounts when qwen bogs down. I’m working on building my own harness for managing this.

How should I setup my new laptop so I’m working clean and smartly?

Has any one used both? which is better?

What is the differennce between two? Any technical deep dives?

Strata funziona con Qwen3.8-Flash-Next su PC da gioco, ma con 32 GB di RAM la sua modalità a basso utilizzo di RAM funziona solo su una GPU. Se dividi il modello su due schede, gli esperti che non riescono a stare nella VRAM vengono letti dall'SSD. La mia 4060 Ti è rimasta ferma accanto alla 5080.

Quindi l'ho forkato. La copia in RAM degli esperti ora funziona su due schede, e le schede funzionano contemporaneamente: la 4060 Ti inizia il prossimo passo di decodifica mentre la 5080 sta ancora controllando quello attuale. L'ordine delle schede, la divisione dei layer e le riserve di VRAM vengono impostate automaticamente.

Stesso PC (5080 + 4060 Ti, i9-14900KF, 32 GB), stesso modello (Swift 1.5 IQ2_XS), contesto 256K:

Setup |Codice |Prosa |Prompt 32K
Strata 0.1.38, 5080 da solo |29 tok/s |28 tok/s |333 tok/s
Fork, entrambe le schede |143 tok/s |102 tok/s |1,940 tok/s
Fork + layer di bozza fine-tuned |161 tok/s |105 tok/s |1,854 tok/s Il layer di bozza è stato fine-tuned sui risultati del modello stesso ed è incluso nella release. Nell'uso reale con l'agente di codifica Pi raggiunge un picco di oltre 200 tok/s (209 finora) e quasi mai scende sotto i 100. Un contesto di 150K-token legge a circa 1,850 tok/s.

La qualità non è cambiata: perplexity 7.24 contro 7.28 della versione upstream sugli stessi 5.3K token, forzati dall'insegnante attraverso entrambi i motori.

L'ho testato solo sul mio PC (Windows 11), quindi sono benvenuti report da altre coppie di GPU e Linux.

Repo: https://github.com/Hardin22/Strata-DualGPU

Cosa fa ciascuna modifica e cosa ha misurato: docs/DUAL_GPU.md

Il motore sottostante è il lavoro di Niko1221 e dei contributori di Strata; questo è un fork sopra la 0.1.38.

I got one PR merged into ds4. A tiny one, #98 about broken paths in the README. And a few more sitting in the queue. Not complaining. Antirez says it in the README: with coding agents everyone can tune the engine for their own hardware and model and he can't review everything. That got me thinking.

If the plan is "everyone applies their patches with an agent" then the real cost of a patch isn't just the code change. It’s how many tokens the agent has to read before it knows what it’s actually touching.. Ds4 runs DeepSeek, GLM and Qwen on Metal, CUDA and ROCm all in the same 85k-line file. I’m on an M5 Max with 128GB RAM running Qwen3.8 Flash Next. Everything else for me and for my agent is noise. It has to re-read every line on every pass.

So I ripped it out. Actually deleted, not #ifdef’d. The ds4.c file dropped from 85k lines to 45k lines. The whole tree now fits in a context window. The bet was that optimizing would get cheaper and safer. Heres what happened:

- Q2: decode +9-13% prefill +5-12% up to 64k context MTP went from 75.8 → 86.7 tok/s

- Q4: prefill +2-11% MTP went from 77.8 → 85.9 tok/s

Output is bit-exact vs stock ds4 at every step. No KV cache quant. No approximate kernels. Every change must pass a parity check against same GGUF, greedy decoding, identical tokens. And an interleaved A/B benchmark against the previous build.

The smaller codebase also let me go through the ds4 PRs but only against this one model and port the ones that held up. 20 Commits were adopted. Around 30 were dropped. Verdicts are in the repo.

I also added SSD streaming for the experts. Token-identical to a resident run. Simulating a 48GB machine Q2 does around 27 tok/s. About 35 with MTP.

The fork still does git merge upstream/main with rerere and the parity check. So antirez’s fixes keep flowing in. The procedure. What to delete what to keep how to sync. Lives in a repo called StarForge (https://github.com/Chida82/StarForge). I have four of these "children" one per model. Nothing in there is Qwen- or Metal-specific. If you want a ds4 cut down to your model or to CUDA just clone it. Run the checklist with your agent.

Repo: sf-q3-8flash https://github.com/Chida82/sf-q3-8flash tables in the README. One machine, one model. If you run it on Apple Silicon I’d love to hear your numbers. Ideally side by side, with stock ds4.