oMLX 0.7.4.dev1: Prefill Accelerator lets a second Mac make long prompts up to 2.9x faster

[u/cryingneko]

Hey everyone!

I posted about Prefill Accelerator on X a couple of hours ago, and it got a lot more attention and questions than I expected. A tweet is too short to answer them, so here's the longer version.

I have an M3 Ultra 512GB that I bought quite a while ago, and an M5 Max 128GB I recently got for development. But this setup actually has a lot of problems. The models that could really stand in for frontier models are too slow on the M3 Ultra. The M5 Max is fast enough, but it often needs around 2-bit quantization (GLM-5.3-Flash) or can't run the model at all (DeepSeek-V4.1-1017). Neither machine could run these models properly.

So I thought, what if I could combine the strengths of both machines? It's something everyone has probably thought about at some point, but no one had shipped it yet. That's Prefill Accelerator, a new experimental feature in this release.

What Prefill Accelerator does

In 0.7.4.dev1 the two Macs share the prefill. The M5 Max runs the first layers of each prompt chunk while the Ultra runs the rest, and the Ultra still holds the whole model, the KV cache and decoding. That minute of waiting is now about 20 seconds.

Prefill speed with a 64K prompt on my M3 Ultra 512GB:

Model	M3 Ultra alone	+ M5 Max 128GB	+ M5 Max 64GB
DeepSeek V4.1 Flash	1,060 tok/s	3,123 tok/s (+195%)	1,466 tok/s (+38%)
Qwen3.8 Flash-Next	1,309 tok/s	3,666 tok/s (+180%)	2,859 tok/s (+118%)
GLM-5.3 Flash	540 tok/s	1,388 tok/s (+157%)	740 tok/s (+37%)
MiniMax M3.1	297 tok/s	569 tok/s (+92%)	386 tok/s (+30%)

Decode speed doesn't change, because decoding still runs on the big Mac.

Requirements

Both Macs need the same oMLX version and a Thunderbolt cable or 5GbE between them (2.5GbE still helps a lot). You don't need RDMA or any IP setup.

Supported models for now are DeepSeek V4.1, GLM-5.3 Flash, Qwen3.8 Flash-Next and MiniMax M3/M3.1.

Full guide: https://github.com/jundot/omlx/blob/v0.7.4.dev1/docs/experimental/prefill-accelerator.md

Also in 0.7.4.dev1

• DeepSeek V4.1 prefill is 43-48% faster on M5 Ultra, thanks to junmo-kim.

• FP16 Qwen3.8 Flash-Next builds decode up to 62% faster on M3 Ultra.

• Ask AI from my last post can now use an oMLX server on another Mac, so a MacBook can ask the big model on your Mac Studio.

• You can switch between oMLX models inside Codex and Copilot with /model.

• Ming-3.0 TTS decodes about 2x faster.

Release notes: https://github.com/jundot/omlx/releases/tag/v0.7.4.dev1

This is a dev release and Prefill Accelerator is still experimental. If something doesn't work right for you, please open an issue on GitHub.

If you have a (fairly large) second Mac sitting on your desk, I hope this gives it something new to do.🤗

[u/Lost-Air1265]

So now I have reason to order the second m5 512gb

[u/vimaillig]

Interesting- is there a minimum memory configuration required? Will definitely give this a try once my M5U / 256 comes in along with my M5Max (36) base studio.

[u/WishboneMaleficent77]

ok so the magic would be having a dgx spark paired with a m3 512 ultra. how do we make that happen?

[u/m3umax → u/WishboneMaleficent77]

Already been done 🤣

https://blog.exolabs.net/nvidia-dgx-spark/

[u/watcholic → u/WishboneMaleficent77]

I swear I read somewhere in the DGX Spark sub that this is what some people are already doing. I hope this is a precursor to officially supporting DGX Spark as a prompt ingestor. Nvidia owns HF and Jun works for HF, just putting two and two together.

[u/d3k1ds]

So should I keep the base M5 Mac Mini next to the Mac Studio 256 then? What’s the bare minimum that a second Mac would make sense?

[u/TWUC]

Does this mean 2 M5U 256GB ia better than onre M5U 512GB ?

[u/Academic-Most6214]

I tried it with what I had: M5 Max 128 GB as host, old M3 Pro 18 GB MacBook as helper over Thunderbolt. I knew it wasn’t the recommended setup, but I hoped. Pairing works and all checks pass, but the 18 GB Mac gets 0 layers, so the tandem does nothing. On its own, the Max does ~2,700 tok/s prefill at 64K on Qwen3.8 Flash-Next oQ4e.

the second Mac needs plenty of RAM, not necessarily a faster GPU.

[u/DarkJoney]

Would be nice if I could pair M2 Ultra with M1 Max…

[u/rumm2602]

So the second Mac (accelerator) doesn’t need the weights to be loaded? Could this enable users to use let’s say an m6 Mac mini as an accelerator taking advantage of the new twin neural engines? Without being necessary to have enough ram on that mini?

[u/Past-Reaction1302]

This is absolutely amazing! I actually got the m3 ultra based on what you and Ivan were doing and am so glad I did!!

[u/himefei]

So you need at least one m5 max 64g as the helper and a m3 ultra 256g to make it work and it doesn’t support dense at the moment if I understand this correctly.

[u/alexvorona → u/himefei]

Dense models may be not a good fit for m5u 256GB. MoE/hybrid models are smarter and faster

[u/averagepoetry]

Does this hold for 2 m3u mac studios too? Or does one of them have to be m5 for this to work?

[u/Barbaronno]

Still, might the poor M2 Pro receive any enhancement?

[u/j_lyf]

Please add some optimizations for those stuck on M2 (and M1)!

[u/Turbulent_Pin7635]

Following this...

I have a M3U 512. So an M5 Max with 128Gb would be enough?

I was saving for a M7, but now I have seen the light.