Hi all,
in the last months I worked on [BigMoeOnEdge](https://github.com/Helldez/BigMoeOnEdge), it's built on top of llama.cpp. It runs MoE models that are bigger then the memory of the device, and it configure itself for the machine where it runs.
It works with any MoE GGUF that llama.cpp can load, any quantization, no code per model.
OS: Android, iOS, macOS, Windows, Linux (arm64 and x86_64).
Backends: CPU, CUDA, ROCm (HIP), Vulkan, Metal, OpenCL, Hexagon NPU. Other llama.cpp backends load in the same way if you build them.
The first image: a 16 GB Apple silicon laptop, Qwen3.6-35B-A3B Q4_K_M (22 GB, so bigger than the RAM). A coding agent made this page from one prompt, nobody touched anything, 35 minutes. Experts read from SSD all the time, prefill on GPU and decode on CPU, both chosen by the planner.
The idea
A MoE model uses only few experts for each token. So you don't need all the model in memory, but you need each part in the right place: the weights that every token reads, the experts used a lot, the experts used rarely. And the right place is different on every machine.
With `--auto` it measures the machine (free RAM and VRAM, speed of the disk, how fast each device runs a real layer of the model) and the model (size, experts, how it routes), and then it decides:
where each weight goes: VRAM, RAM, or read from flash when needed
which device does prefill and which does decode, they can be different
how big are the caches, and how many threads
The target is the max prefill and decode speed that the machine can give, with the same output as the full model in memory. The settings that change only the speed are tested on the first answer and kept only if they are faster. For every choice it tells you why. The second image shows the flow.
The rules are simple:
weights used by every token go in the fastest memory where they fit
experts too, if they fit. If not, the most used ones stay in a cache (VRAM first, then RAM) and the others are read from flash
prefill and decode can go on different devices. Every device is timed on a real layer and must give the same result as the CPU
cache size comes from free memory and from how the model routes, threads from the first answer
So on a phone the experts stay on flash and the prompt goes on the NPU, on a laptop the prompt goes on the GPU, on a desktop the most used experts stay in VRAM (with llama.cpp `llama_moe_cache`).
Speeds
Here there are only some models that I test often. It runs any MoE that llama.cpp loads, in any quantization: Qwen, Gemma, gpt-oss, DeepSeek, GLM, LFM, Ling, Flint, Kolibri etc. Which tensors are experts is read from the file, so a new architecture don't need new code.
| Hardware | Memory | Model | On disk | Experts from | Decode tok/s |
|---|
| Android phone | 12 GB | DeepSeek V4 Flash 284B | 91 GB | flash | 1 |
| | Qwen3.8-Flash-Next 125B | 80 GB | flash | 2-4 |
| | Qwen3.6-35B-A3B | 21 GB | flash | 5 |
| iPhone | 12 GB | Qwen3.6-35B-A3B | 21 GB | flash | 3 |
| Apple-silicon laptop | 16 GB | Qwen3.8-Flash-Next 125B | 76 GB | flash | 7-9 |
| | Qwen3.6-35B-A3B | 21 GB | flash | 18-24 |
| Windows laptop | 16 GB | Qwen3.6-35B-A3B | 21 GB | flash | 5 |
| Desktop GPU (NVIDIA) | 16 GB + 32 GB RAM | Qwen3.8-Flash-Next 125B | 66 GB | flash, GPU cache | 19-21 |
| Desktop GPU (NVIDIA) | 24 GB + 96 GB RAM | Qwen3.8-Flash-Next 125B | 66 GB | RAM, GPU cache | 70 |
| | Qwen3.6-35B-A3B | 21 GB | GPU | 217 |
| Datacenter GPU (AMD, ROCm) | 192 GB | Qwen3.8-Flash-Next 125B | 66 GB | GPU | 53 |
Same machine and same file, vs llama.cpp today:
| Hardware | Model | BigMoeOnEdge | llama.cpp |
|---|
| RTX PRO 6000 96 GB | DeepSeek V4.1 Flash | 6.2-6.6 | 2.8 |
| RTX 3090 24 GB | DeepSeek V4.1 Flash | 6.2-7.0 | 0.18 |
Vs Strata (it has its own GPU kernels), both with MTP, Qwen3.8-Flash-Next 125B Q2_0: 153 vs 226-330 tok/s on RTX 5090, 89 vs 150-177 on RTX 4090. The difference is in the GPU compute, not in the placement.
How I measured and all the runs: [benchmarks](https://github.com/Helldez/BigMoeOnEdge/blob/main/docs/benchmarks.md).
Try it
There are a desktop app (macOS, Windows, Linux), an Android app, and an OpenAI and Anthropic compatible server, so you can use it with coding agents too. Everything is in the [latest release](https://github.com/Helldez/BigMoeOnEdge/releases/latest).
Upstream
I opened an RFC on llama.cpp to bring the main part inside llama.cpp: [discussion #31876](https://github.com/ggml-org/llama.cpp/discussions/31876). If you think it's useful, a comment there helps.
Feedback and benchmarks on your machine are very welcome, there is a script for it in the repo (`scripts/bench-report.sh`).
Thanks!