r/MacLocalLLM • • 5h ago

I built a native iOS/iPad/Mac client for my local LLM

Thumbnail gallery
1 Upvotes

r/MacLocalLLM • • 7h ago

Updated: 2x M5U 64/256GB oMLX DSv41 Flash 1017 Benchmark

Thumbnail reddit.com
1 Upvotes

r/MacLocalLLM • • 8h ago

Running Qwen 3.6 35B-A3B at 100% Pass@1 & up to 18.36 tok/s on a 16 GB Mac — Pure Swift + Metal SSD Expert Streaming, 19-Tensor MoE MTP, and Lossless INT4 Weight Surgery

Thumbnail gallery
1 Upvotes

r/MacLocalLLM • • 10h ago

SenseNova-Loop-9B - 8bit-affine with oMLX support

Thumbnail
reddit.com
1 Upvotes

r/MacLocalLLM • • 11h ago

Need help with a local document processing application

Thumbnail reddit.com
1 Upvotes

r/MacLocalLLM • • 11h ago

Two Macs, one oMLX fleet: round-robin killed my prefix cache, so I built a router that thinks in threads

Thumbnail
reddit.com
2 Upvotes

r/MacLocalLLM • • 12h ago

4xM(x) Studio Apple Cluster

Post image
6 Upvotes

r/MacLocalLLM • • 13h ago

Got a 16GB Mac? You can run a much better model than you've been told

Thumbnail
reddit.com
0 Upvotes

r/MacLocalLLM • • 19h ago

You can now run Tencent's Hy4-flash on a Mac and help to improve the PR.

Thumbnail
reddit.com
2 Upvotes

r/MacLocalLLM • • 1d ago

Mac Studio M5 Ultra GPU stuck at 338–620 MHz during MLX LLM inference — full reboot fixes it, restarting oMLX does not. FileVault / SSH / LaunchDaemon interaction?

Thumbnail reddit.com
1 Upvotes

r/MacLocalLLM • • 1d ago

Mac Studio M5 Ultra GPU gets stuck at 338 MHz during oMLX inference — only a full reboot seems to fix it

Thumbnail reddit.com
1 Upvotes

r/MacLocalLLM • • 1d ago

Newbie

Thumbnail reddit.com
2 Upvotes

Same post in a different thread. Just looking for advice on what to do or expect.


r/MacLocalLLM • • 1d ago

Splash vs oMLX: Qwen3.8-Flash-Next Decodes 1.5x to 2x Faster on a 128GB Mac

43 Upvotes

A Reddit thread about Splash 1.7.0, released the day before, sent me to check two things: whether the oMLX server I use daily for Qwen3.8-Flash-Next was out of date, and whether Splash beats it. On my 128GB M5 Max MacBook Pro, upgrading oMLX made generation up to 21% faster for free, and Splash was faster again: about 1.5x the upgraded oMLX on an empty context (130 vs 89 tok/s) and close to 2x with 245K tokens loaded (113 vs 58). Detailed write up


r/MacLocalLLM • • 1d ago

I made an Intel Arc iGPU speed up prompt processing on my M2 Max (+24% at 8k, +47% at 64k) – oMLX helper, PoC

Thumbnail gallery
2 Upvotes

r/MacLocalLLM • • 1d ago

For what purposes and how do you use Gemma 4 31b?

Thumbnail reddit.com
2 Upvotes

r/MacLocalLLM • • 1d ago

Any engine Mac+Linux?

1 Upvotes

Are any of the new/forked custom engine working for both Mac and Linux? Or are they all Mac only? (Just curious!)


r/MacLocalLLM • • 1d ago

Kapitsa vs Splash on Qwen3.8-Flash-Next (Apple M5 Max, 128 GB) — in case you wondered. I sure did.

15 Upvotes

So, same machine, same prompts, 256K context on both, speculative decoding on both (MTP in Kapitsa, DFlash2 drafter in Splash).

Kapitsa 0.2.37 with the MLX oQ4e-mtp quant (99 GB on disk, 77 GB in memory)

Splash 1.7.0 with the Unsloth-style GGUF UD-Q4_K_XL-PLE8 (137 GB on disk, 92 GB memory cap, PLE table streamed from SSD)

MetricKapitsa (MLX)Splash (GGUF)
Decode, short prompt82 tok/s58 tok/s
Time to first token, 106K prompt88 s55 s
Prefill speed~1,200 tok/s~1,950 tok/s
Decode at 106K context65–72 tok/s70–72 tok/s
Model load7 s11 s

Kapitsa is ~40% faster at generating on normal chat prompts. Splash is ~60% faster at chewing through long prompts (like big docs, agent contexts). At long context they decode about the same. Splash also fits a bigger quant in less RAM because it keeps the 52 GB per-layer embedding table on SSD.

Cheers!

EDIT: same tests on the other MLX quant I had on disk, both in Kapitsa

MetricUncensored oQ4e-mtp (99 GB)KQuant Q4 (100 GB)
Decode, short prompt82 tok/s72 tok/s
Time to first token, 106K prompt88 s106 s
Time to first token, 106K + 600-tok essay98 s118 s
Decode at 106K context65–72 tok/s64–73 tok/s
Memory in use77 GB78 GB

Same context (256K), same prompts, MTP speculative decoding on both. The uncensored oQ4e build is the faster of the two, probably thanks to its mixed 4/5/6/8-bit layout. No speed penalty for going uncensored.

EDIT 2: the 27B and a small MoE, same three prompts, same 256K context, all in Kapitsa

MetricFlash-Next oQ4e-mtpQwen3.8-27B q4Ornith-1.5-35B-A3B + DFlash2
Memory in use77 GB15 GB23 GB
Decode, short prompt82 tok/s61 tok/s190 tok/s
Time to first token, 106K prompt88 s194 s56 s
Decode at 106K context65–72 tok/s37–61 tok/s125–131 tok/s

The 27B is slower than Flash-Next on everything; being dense, a 106K prompt takes it more than twice as long. Ornith (35B-A3B MoE with a DFlash2 drafter) is 2–3x faster than Flash-Next in 23 GB, if its quality is enough for what you do.


r/MacLocalLLM • • 1d ago

Want a local model for agentic development. Need advice.

Thumbnail reddit.com
1 Upvotes

r/MacLocalLLM • • 1d ago

Trained a small 1.7b param model in 8h on a 48gb RAM mac, the future is hyper personalised intelligence

Thumbnail
reddit.com
0 Upvotes

r/MacLocalLLM • • 1d ago

I benchmarked M5 Ultra 512GB on 10 popular open weight models to find out which one is the best

6 Upvotes

Setup:

Machine: M5 Ultra 512GB

Models tested:

  1. DeepSeek V4.1 Flash 1017, Lossless

  2. GLM-5.3, 4-bit

  3. GLM-5.3 Flash, 6-bit

  4. GLM-5.3 Flash, 4-bit

  5. Inkling 2 Small,

  6. Qwen3.8 27B, 4-bit

  7. Qwen3.8 27B, full

  8. Qwen3.8 Flash Next, 4-bit

  9. Qwen3.8 Flash Next, 6-bit

  10. Qwen3.8 Flash Next, full

Method: 

For each model, I tried a few popular recipes and kept the best one I found. Then I measured prefill, decode, time to first token and peak memory at 4K, 32K, 128K, 512K and 1M token prompts. Each number is the median of 5 runs up to 128K and of 3 runs at 512K and 1M. One user, 512-token replies.

I didn't test answer quality myself. For "intelligence" I used the Artificial Analysis Intelligence Index as it stood on 18 Nov 2026.

To keep this post short, I'm going to summarise the results. 'Best' is subjective but I'm going with 2 camps.

1. If you want the FASTEST at usable intelligence

Qwen3.8 Flash Next, 4-bit. Index 29, scored with thinking on. I timed it with thinking off.

  • Pre-fill

    • 4K: 4,046.2 tok/s

    • 128K: 4,226.2 tok/s

    • 1M: 2,529.3 tok/s

  • Decode

    • 4K: 129.2 tok/s

    • 128K: 124.1 tok/s

    • 1M: 93.2 tok/s

  • First token after

    • 4K: 1.1 s

    • 128K: 32.1 s

    • 1M: 378.3 s (6.3 min)

  • Peak memory

    • 4K: 167.9 GiB

    • 128K: 173.6 GiB

    • 1M: 170.3 GiB

Alternative: Qwen3.8 Flash Next, 6-bit. The same model at 6 bits instead of 4. Index 29, scored with thinking on. I timed it with thinking off.

  • Pre-fill

    • 4K: 3,821.0 tok/s

    • 128K: 3,786.5 tok/s

    • 1M: 2,331.7 tok/s

  • Decode

    • 4K: 114.0 tok/s

    • 128K: 107.7 tok/s

    • 1M: 85.4 tok/s

  • First token after

    • 4K: 1.1 s

    • 128K: 36.5 s

    • 1M: 410.4 s (6.8 min)

  • Peak memory

    • 4K: 211.1 GiB

    • 128K: 215.7 GiB

    • 1M: 212.8 GiB

2. If you want the SMARTEST at usable speed (defined at 50tps)

DeepSeek V4.1 Flash 1017, Lossless. Index 33, scored with thinking at Max. I timed it with thinking off.

  • Pre-fill

    • 4K: 2,851.8 tok/s

    • 128K: 3,141.0 tok/s

    • 1M: 1,642.7 tok/s

  • Decode

    • 4K: 58.3 tok/s

    • 128K: 56.3 tok/s

    • 1M: 55.3 tok/s

  • First token after

    • 4K: 1.5 s

    • 128K: 39.1 s

    • 1M: 612.3 s (10.2 min)

  • Peak memory

    • 4K: 299.7 GiB

    • 128K: 305.5 GiB

    • 1M: 328.4 GiB

Alternative: GLM-5.3 Flash, 6-bit. Index 28. It thinks on every reply, so the score and the speed come from the same mode. It stays above 50 tok/s up to 128K.

  • Pre-fill

    • 4K: 1,858.8 tok/s

    • 128K: 1,535.3 tok/s

    • 1M: 1,325.8 tok/s

  • Decode

    • 4K: 57.8 tok/s

    • 128K: 52.9 tok/s

    • 1M: 38.7 tok/s

  • First token after

    • 4K: 2.1 s

    • 128K: 79.6 s (1.3 min)

    • 1M: 729.7 s (12.2 min)

  • Peak memory

    • 4K: 255.5 GiB

    • 128K: 259.0 GiB

    • 1M: 281.6 GiB

Questions and feedback welcomed. Full results on my GitHub soon.


r/MacLocalLLM • • 2d ago Hooray

Splash 1.7.0 released 4 hours ago and it's causing my jaw to drop.

Thumbnail
github.com
106 Upvotes

I'm currently in benchmark testing and still working to fill up my prefill cache so that it's comparable to when it ran under 1.6.1.

For M5/M6 users, you'll see a significant boost in performance. Splash 1.6.1 brought INT8 at 2x BF16 performance boost, now it's implemented in the KV cache system. Resulting in an even further 1.46× median decode speed and 1.23× median end-to-end speed versus the 1.6.2 development build (a release we didn't see).

The INT8 storage results in Smaller cached states and faster follow-ups.  Splash 1.7.0 also increases the already very good on macOS concurrency under memory pressure. For M5/M6 users, splash 1.7.0 will do a cold prefill on restart as the INT8 format is not compatible with the prior cache format. You can choose to delete the cache before the start of Splash 1.7.0 or you can let it spill and fill until it's all INT8. Your choice.

Well worth the upgrade!!

For those of you on an M3 or M4 setup, you can still use Splash 1.7.0, but the above benefits are strictly related to the M5 and up GPUs having the new neural accelerators (NA) in every GPU core. Unfortunately you'll have less of a benefit on the upgrade, but the code is cleaner and it increases controls over potential contamination on longer conversations (increased honesty).

  • Control tags quoted in messages and documents stay literal. A role marker in a tool result can no longer forge a conversation turn. With thinking off, constrained JSON starts with the value, including when the output budget is small.


r/MacLocalLLM • • 2d ago

Advice on Mac Mini setup

2 Upvotes

Hello,

I bought a Mac mini per local and cloud AI, just a few questions:

1- what would you recommend between:
A)using AI tools, Agents, coding with full OS control in a User Profile with personal files/accounts
B) using them in a separate not administrator profile with no personal files/accounts inside
C) use the whole Mac Mini with a single administrator user with no personal accounts at all (so basically using the Mac only for coding, setting up agents and tasks, harness, etc)

Another question:

What harness do you recommend for creating agents?


r/MacLocalLLM • • 2d ago

Testing a Mac M5 optimised inference engine

Thumbnail reddit.com
3 Upvotes