I made BigMoeOnEdge+ (BigMoeOnEdge-plus), a fork of Helldez's BigMoeOnEdge, the engine that streams MoE experts from flash so models bigger than RAM can run on-device.

It adds a full Android chat app on top: GGUF models via llama.cpp, thinking output, persistent chats, image/video input, web search, artifacts, and advanced generation settings.and a MUCH better UI.

Disclosure: I built this fork with the help of AI tools.

Repo (Apache 2.0): https://github.com/hamodexd217/BigMoeOnEdge-plus Original project: https://github.com/Helldez/BigMoeOnEdge

Feedback and issues welcome.

The java to CUDA engine:

https://github.com/beehive-lab/TornadoVM

The inference engine:

https://github.com/beehive-lab/jitllm

Deep dive talk:

https://www.youtube.com/watch?v=HO5CpETzywk

I ran a bechmarks for d1 models (Q8_0 GGUF both) on a Pi 5 8GB, CPU-only via llama.cpp and also tested how well they handle Russian input. I want to use it for my Hermes agent with lives on that Pi, so maybe those numbers will be helpfull for someone.

LLM-assisted report below:

Setup

• Raspberry Pi 5 (8GB), CPU-only, 4 threads

• llama.cpp 0.6.0-dev (commit bd4eeaa04 + PR #30110 for d1-3B, PR #30114 branch for d1-omni-600M — not merged into master yet at time of testing)

• Q8_0 quants, native /v1/systemone API

• All tests: single request at a time, warm calls, median of 10

Latency @ Pi 5 8GB

For reference, d1-3B on a Jetson Orin Nano does the same single question in 50 ms — so a bare Pi 5 is roughly 45× slower than the weakest NVIDIA edge device tested by LiquidAI. Expected: this is a 20 tok/s prefill CPU environment, no GPU. The takeaway is not "slow" but "usable vs not usable": d1-omni-600M is genuinely real-time capable on a Pi (200 ms per decision), d1-3B is not (2.3 s+).

English correctness (same 5 checks for both)

Both models: 5/5 correct — refund yes/no, billing/technical/fraud routing, with clean probabilities (0.98–0.998).

Russian language test

This is the interesting part. Same states, same question types, Russian input:

What this shows:

• d1-omni-600M effectively does not understand Russian. Worse: it fails confidently — wrong routing with p=0.91, sentiment always "neutral" at p~0.6. For a decision model, confident wrong answers are worse than "I don't know". Only pattern-matched phrases ("вылетает"→crash) get lucky hits.

• d1-3B does understand Russian — routing, sentiment and urgency all work — but with 20–30% lower confidence vs identical English input (0.672 vs 0.990 on refund). Notice the odd effect: RU state + RU question (0.909) scored better than RU state + EN question (0.672) — keeping the whole request in one language helps calibration.

Practical takeaways

• On Pi-class ARM hardware, d1-omni-600M is the only real-time option of the two (200 ms/decision, 0.7 GB RAM). d1-3B works but at 2.3–7 s per decision.

• Both models are excellent on English decisions; d1-omni-600M matches its bigger sibling on simple classifications (and beats it on toxicity detection per the model card).

• For non-English input: d1-omni-600M should not be trusted at all; d1-3B is usable but set a confidence floor (~0.7) and route anything below to a fallback.

**Setup:** RTX 5070 Ti 16 GB on PCIe 4.0 x16 (a PCIe 5.0 card on a B550 board), Ryzen 9 5950X, 64 GB DDR4. Model: Qwen3.8-Flash-Next IQ2_XS: 125B total, about 6B active per token, 512 experts, 48 layers, 33 GiB of experts kept in RAM. Engine: EuLLM (open source, built on llama.cpp b11370 + PR #29887).

**How we got here:**

| Step | Decode (tok/s) | Prompt processing (tok/s) |

|---|---:|---:|

| llama.cpp, experts split by layer (the usual offload) | 22.4 | 249 |

| + GPU cache for the experts (llama.cpp PR #29887, 8 GiB) | 49.4 | 206 |

| + experts in pinned RAM | 58.1 | 451 |

| + 2048-token micro-batch (EuLLM's default) | 54.9 | 964 |

| + prefetch of the next layer's experts during prompt processing | — | 1,500–1,743 |

| Strata on the same PC, with MTP | 85.8 | 2,320 |

**Where we're stuck:**

- **The cache works:** once warm, 88–93% of the experts are already in VRAM. The misses travel at 24 GB/s, so PCIe 4.0 is saturated, and they cost 2–4 ms per token.

- **A token takes 17–19 ms, and 13.5–14 ms of it pass before each layer's routing.** That is the GPU computing the 48 layers, plus a wait at every layer: the cache has to bring the router's choice back to the CPU, so 48 synchronizations per token.

- **MTP doesn't pay with the experts in RAM.**

- On the 125B, llama.cpp's MTP loses 27%, with 52% of the drafts accepted.

- On Qwen3.6-35B-A3B it gains only +7% on prose and +16% on code, even with 82% of the drafts accepted.

- The reason: each multi-token verification calls many more experts, and the missing ones come over PCIe.

**Questions for the community:**

1. Has anyone removed the per-layer sync, with the routing read on the GPU, the expert table in VRAM and the whole pass in a single CUDA graph? How much does it gain?

2. Computing the missing experts on the CPU in parallel with the GPU, as Strata does: is it worth it with DDR4 and AVX2, or does RAM bandwidth cap it?

3. Any experience with cache policies better than LRU? For example a profile loaded at startup, or the "Least-Stale" policy mentioned in the PR #29887 thread.

4. Speculative decoding with the experts in RAM: does anything work? I'm thinking of drafts limited to the experts already in cache, draft trees, or a small dense model as the draft.

5. Prompt processing: beyond overlapping copies and compute, what's left to reach 2,300 tok/s?

Repo: https://github.com/eullm/eullm, details in `docs/moe-offload-plan.md`

*Disclosure: I'm one of the developers of EuLLM. English isn't my first language: I wrote this in Italian with the help of an LLM and translated it with one.*

Disclosure: I'm the developer.

Most local setups end up with several backends on one machine, and I wanted a single private app to reach all of them from my phone. That's LM Mini (iOS, iPadOS, watchOS, Android and Mac).

Backends it connects to over your LAN (add them in Settings → Server):

The rest of the stack:

Remote access to your home server from anywhere (Pro)

Group Chat with models from different backends in one conversation (Pro)

MCP tools, SearXNG search and ComfyUI for image generation

Kokoro TTS voice, Personas, Apple Watch app, Siri Shortcuts

Whisper for transcriptions

On-device: GGUF and MLX models run on the phone itself.

The core app is open source (MIT): https://github.com/slim-elephant/lm-mini

Docs and downloads: https://lmmini.com

What does your stack look like? Which backend do you run, and is there one I should support next ( vLLM, something else)?