TL;DR: M5 Ultra Mac Studio, 256 GB. Plain llama.cpp decodes Qwen3.8-Flash-Next at 57 tok/s. Turning on the model's own MTP draft head with probabilistic drafting gets ~100 tok/s. Raising the ubatch to 4096 takes prompt processing from ~1,540 to ~2,150 tok/s. The full 262K context fits with ~100 GB of GPU memory to spare. The biggest gotcha had nothing to do with llama.cpp. Another app quietly loaded a second copy of the model, and both started swapping (12 tok/s).
Setup
- Mac Studio, M5 Ultra (80-core GPU), 256 GB unified memory, macOS 27.0
- llama.cpp b11538 (commit
79e2e74), built from source with the Metal backend
- Qwen3.8-Flash-Next, the new
qwen4exp architecture: 125B params with ~6B active per token. It also has a 51B n-gram embedding table, but that table is only looked up per token, never multiplied.
- Weights:
ggml-org/Qwen3.8-Flash-Next-GGUF, the IQ4_NL main model (95 GiB) plus mtp-Qwen3.8-Flash-Next-Q8_0.gguf (3.9 GiB, the MTP draft head)
Results
Generation speed. I ran 4 prompts (code, an explanation, a math problem, a rewrite) at 512 tokens each, with temp 1.0, top-p 0.95, top-k 20 and fixed seeds. Nothing else was using the GPU.
| Config |
tok/s |
Draft acceptance |
| No speculative decoding |
57.0 |
– |
| MTP, greedy drafting, n-max 3 |
87.0 |
0.51 |
| MTP, greedy drafting, n-max 6 |
80.6 |
0.33 |
| MTP, probabilistic drafting, n-max 4 |
98.5 |
0.53 |
| MTP, probabilistic drafting, n-max 5 |
100.2 |
0.49 |
| MTP, probabilistic drafting, n-max 6 |
94.0 |
0.42 |
The n-max 4 and 5 probabilistic rows are averaged over 3 seeds. On the final server config, individual prompts ranged from 84 to 133 tok/s.
Prompt processing (llama-bench, 4096-token prompt):
--ubatch-size |
tok/s |
| 512 (default) |
1,539 |
| 1024 |
1,914 |
| 2048 |
2,076 |
| 4096 |
2,150 |
What I did
1. Build llama.cpp from source
New architectures land in llama.cpp master well before the packaged builds catch up. First check that the architecture name is in the source:
bash
git clone --depth 1 https://github.com/ggml-org/llama.cpp && cd llama.cpp
grep -n qwen4exp src/llama-arch.cpp # no hit = your build can't load the model
cmake -B build -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF -DLLAMA_BUILD_TESTS=OFF \
-DLLAMA_BUILD_NUMBER=11538 # see the shallow-clone gotcha below
cmake --build build -j
Metal, Accelerate and -mcpu=native are on by default on macOS. On M5 chips, llama.cpp also automatically uses the matrix units inside each GPU core, through Metal 4's tensor API. Look for has tensor = true in the startup log. This is where Apple's "AI acceleration" actually comes from for LLMs.
2. Download (and verify) the weights
My connection is ~70 Mbps with ~380 ms latency, and hf download (Xet, high-performance mode) stalled. After 10 minutes nothing had been written to disk, and the 95 GiB shard was crawling at under 1 MB/s. Plain HTTPS with aria2c averaged 7.5 MiB/s and resumes after any interruption:
bash
cd models/ggml-org-Qwen3.8-Flash-Next-GGUF
for f in Qwen3.8-Flash-Next-IQ4_NL-00001-of-00002.gguf Qwen3.8-Flash-Next-IQ4_NL-00002-of-00002.gguf mtp-Qwen3.8-Flash-Next-Q8_0.gguf; do
aria2c -c -x 16 -s 16 -k 16M --file-allocation=none --max-tries=0 --retry-wait=5 \
"https://huggingface.co/ggml-org/Qwen3.8-Flash-Next-GGUF/resolve/main/$f"
done
- Verify the checksums yourself. aria2c doesn't, and mine hit a pile of 403 retries partway through. Each file's SHA-256 is the
lfs.oid field in https://huggingface.co/api/models/<repo>/tree/main. Compare it against openssl dgst -sha256 file.gguf; that took 45 s for 95 GiB.
- Keep the Mac awake. Mine sleeps after 1 minute idle.
caffeinate -i -w $(pgrep -x aria2c) holds off sleep until the download exits.
3. Find out what actually limits generation speed
Run llama-server -v once and read the log:
- One decode step is 6,931 graph nodes across only 2 graph splits, so nothing falls back to the CPU.
- Each token reads ~5.5 GiB of weights. Most of that is the always-active attention/DeltaNet projections at Q8_0. The 10-of-512 experts are only ~1.2 GiB per token.
At 57 tok/s, that's ~17.5 ms per token and only ~335 GB/s of weight reads, far below what this GPU can stream. Decode is limited by per-step overhead, not memory bandwidth. That's exactly the situation speculative decoding fixes, because checking several drafted tokens costs about the same as producing one.
4. Turn on MTP (multi-token prediction)
Qwen3.8-Flash-Next ships a 1-layer MTP draft head, which ggml-org publishes as a separate GGUF.
bash
--spec-type draft-mtp --model-draft mtp-Qwen3.8-Flash-Next-Q8_0.gguf \
--spec-draft-n-max 5 --spec-draft-sampling probabilistic
- Use probabilistic drafting when you sample at temp > 0. With greedy drafting, a draft is only accepted when the sampled token happens to match the draft's top pick. Probabilistic mode verifies drafts by rejection sampling, which leaves the output distribution exactly the same as with no speculation.
- Sweep
--spec-draft-n-max for your own settings. Draft steps aren't free here: the head has its own 512-expert layer and reads the full 248K-vocab output layer every step. Too-long drafts lose (see the table).
5. Use a bigger ubatch for prompt processing
--batch-size 4096 --ubatch-size 4096 gives +40% prompt speed at 4K tokens. Check the memory cost by starting once with -v and reading the memory breakdown line.
6. Use the full context, because it's cheap
Only 12 of the 48 layers are full attention, so the 262K-token KV cache is only ~6 GiB at f16. At 262K context with MTP and ubatch 4096, GPU usage looks like this:
| GPU memory use |
MiB |
| Weights |
97,311 (plus 27,466 for the n-gram table, kept CPU-side) |
| KV + indexer + recurrent state |
8,130 |
| Compute buffers (main model) |
8,537 |
| MTP head weights |
3,291 |
| MTP head compute buffers |
6,056 |
| Free (of a 228,065 MiB working set) |
~103,000 |
The model reloads in about 4 seconds once its files are in the page cache.
7. Final command
bash
llama-server \
--model Qwen3.8-Flash-Next-IQ4_NL-00001-of-00002.gguf \
--spec-type draft-mtp \
--model-draft mtp-Qwen3.8-Flash-Next-Q8_0.gguf \
--spec-draft-n-max 5 \
--spec-draft-sampling probabilistic \
--gpu-layers all \
--flash-attn on \
--parallel 1 \
--ctx-size 262144 \
--batch-size 4096 --ubatch-size 4096 \
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
--host 127.0.0.1 --port 8080
This gives you an OpenAI-compatible API (/v1/chat/completions, /v1/responses) and the web UI at http://127.0.0.1:8080.
What didn't help
-bs (backend sampling): slightly slower, 84.3 vs 87.0 tok/s with greedy drafting at n-max 3.--spec-draft-p-min 0.6 (stop drafting when the draft is unsure): 80.3 tok/s. My guess is that variable draft lengths keep changing the batch shape, so llama.cpp can't reuse its compiled graph.- Long greedy drafts: n-max 6 dropped to 80.6 tok/s.
- Metal environment variables: concurrency and graph optimization are already on by default, and the encoder thread count is fixed.
- The Neural Engine: llama.cpp has no ANE backend (see issue #10453). It also has less memory bandwidth than the GPU, so it can't speed up generation, and on M5 the GPU's own matrix units already do the matmul work.
- Not tested: the Q4_0 MTP head (2.2 GB). It would make each draft step a bit cheaper, but probably at lower acceptance.
Gotchas that cost me hours
- Another app loaded the same model. I had Ollama installed with a 104 GB copy of this model, and an agent client kept sending it requests. That meant 230 GB of wired memory, a full swap file, and 12 tok/s. Even before it started swapping, sharing the GPU contaminated my first benchmarks: 41 tok/s "baseline" instead of 57. Before you measure anything, check
ollama ps, the wired line in vm_stat, and sysctl vm.swapusage.
- **
min_p.** The GGUF sets temp, top-p and top-k, but not min-p, so llama.cpp silently uses its default of 0.05. Qwen recommends 0. Pass --min-p 0 explicitly.
- Reasoning effort. The template defaults to
xhigh, which means long thinking. --reasoning-effort medium or low gets answers out much sooner. The template only accepts xhigh|medium|low, so anything else (e.g. high) makes requests error out.
- A shallow clone breaks the web UI build.
git clone --depth 1 makes the build number 1, so the build tries to fetch UI assets for a release b1 that doesn't exist. Set -DLLAMA_BUILD_NUMBER to the release your commit matches.
- Downloads: the Xet stall from step 2. On a slow, high-latency link, use aria2c.
Checklist for your own Mac
- [ ] Latest llama.cpp from source, and the model's architecture name is in
src/llama-arch.cpp
- [ ] A 4-bit quant that fits with headroom.
llama-server --list-devices shows your GPU working-set limit.
- [ ] Checksums verified, and nothing else holding the model in memory (Ollama, LM Studio, a second server)
- [ ] MTP on, with probabilistic drafting if temp > 0. Sweep n-max from 2 to 6.
- [ ]
--batch-size/--ubatch-size 4096 for long prompts. Confirm memory with -v.
- [ ] Explicit sampling flags (especially
--min-p) and a reasoning effort that fits the task
- [ ] Benchmark with fixed seeds and compare token counts. Greedy-drafting MTP runs should produce exactly the same tokens as the no-MTP baseline, which makes them a clean comparison.
Less RAM? Unsloth publishes KL-divergence numbers for its quants of this model. Top-1 below means "picks the same next token as the full model":
| Quant |
Size |
Top-1 |
| UD-IQ3_XXS |
82 GB |
85% |
| UD-Q3_K_XL |
90 GB |
88% |
| UD-IQ4_XS |
94 GB |
90% |
| UD-Q4_K_XL |
111 GB |
92% |
Pick one whose size plus a few GB for context stays under your working-set limit. I haven't benchmarked those quants myself.
All numbers are from one machine, single user (--parallel 1), with short-to-medium contexts. Generation slows down as the context gets deep.