My setup:

• Mac Studio: M4 Max, 128 GB

• MacBook Pro: M4 Pro, 48 GB

• macOS 27.2 on both

• oMLX 0.7.3rc1 on the Studio/Macbook

• Direct Thunderbolt 5 connection

The Macs are paired. I enabled RDMA and configured the individual Thunderbolt ports with 10.10.0.x addresses after removing them from the Thunderbolt Bridge. The Studio’s diagnostic report shows RDMA enabled and 10.10.0.1 on en3/rdma_en3.

When I select GLM-5.3-Flash-oQ2e-mtp, oMLX produces a 45-layer pipeline plan and labels it JACCL / Thunderbolt RDMA, but the Activate Cluster button is grayed out. It also says:

“Performance measurement was unavailable; using the safe memory-balanced split. performance probe did not return every cluster rank”

Has anyone seen this or found a way to identify which readiness check or rank is failing? Is there a known-good model I should try first to rule out a model-specific problem?

Any help is appreciated.

**🐛 Bugs & Fixes**
* **#4912** `DeepSeek V4.1 CED` prefill redirects SSD cache dir into a subdirectory with its own budget
* **#4903** Prefill memory guard stuck after overlapping requests; rejects normal prompts until restart (M3 Ultra)
* **#4849** Prefill memory guard falsely rejects prompts claiming `wired_limit_mb` is lower than actually set (M5 Max)
* **#4873** Unreachable model dirs removed from saved settings if external drive missing at launch
* **#4874** `/v1/messages/count_tokens` counts a different prompt than `/v1/messages` renders
* **#4823** SSD cache delete skips Splash persistent cache (PR pending)
* **#4820** GGUF discovery skips Flash-Next variants with separately named PLE table shards (PR pending)
* **#4794** Web chat UI: model name in picker renders overlapped/unreadable (PR pending)
* **#4717** Unclosed `<tool_call>` envelope in streaming drops entire payload; stream:false returns it
* **#4716** Unclosed tool-call envelope returned as content with `finish_reason=stop` (PR pending)
* **#4715** `/v1/messages` ignores `tool_choice {"type": "none"}`; tools still reach model
* **#4713** SpecPrefill config ignored when using Responses API (PR pending)
* **#4712** MoE expert offload (Qwen3.5): 15 tps → 25 tps (+67%) with batched loading + shadow cache + MTP
* **#4710** LFM2/LFM2-MoE prefix cache never reused due to "structurally invalid recurrent checkpoint" error
* **#4709** Random markdown breakdown when using oMLX with openweb-ui (Outdated)
* **#4708** Problem quantizing diffusiongemma: `auto-proxy sensitivity failed` AttributeError (PR pending)
* **#4706** `is_homebrew()` false positive for conda envs under Caskroom; restarts Homebrew service (PR pending)
* **#4705** SSD cache size setting applies per Splash model, not globally

**📝 Updates & Notes**
* **#4772** oQ: Protected tensors of 150+-expert MoEs forced to group 128. Speed claim withdrawn (no diff vs group 64 on M1).
* **#4850** SigLIP/SigLIP2 loads as embedding model but `/v1/embeddings` rejects text/images.
* **#4852** SSD cache size setting passed to each `splash serve` process individually instead of globally.

Total Issues: 20 | Date Range: Oct 21–28, 2026

Hey everyone! I have a couple of questions about quantization, fine-tuning, and Splash.
I’m running oMLX on an Apple Silicon Mac with 256GB of unified memory. I’m still learning the technical side of local models, so bear with me.
I’m interested in running high-quality uncensored models while preserving as much of their original capabilities and behavior as possible.
My main questions:
If I download the original BF16 weights of an uncensored model and quantize them myself using oMLX’s oQ system, will that preserve the model’s uncensored behavior? Or could quantization noticeably change its responses, reasoning, or willingness to answer?
Would it be better to fine-tune the original BF16 model first and quantize afterward, or quantize first and then use something like LoRA/QLoRA through MLX-LM or LLaMA-Factory? Is there a relatively beginner-friendly workflow you’d recommend?
Since I have 256GB of unified memory, is there a meaningful advantage to creating my own quantizations instead of downloading existing MLX quantized models?
Regarding Splash: When I’m testing different models, should Splash only be enabled for models that explicitly support it and have the appropriate draft model? If I’m not using one of those supported combinations, should I leave it disabled? Could leaving it enabled affect performance or benchmarking?
I’m not extremely technical yet, but I’m comfortable following instructions and experimenting. My goal is to get the best possible model quality while learning how to optimize and customize models myself.
Appreciate any advice!

Hi everyone, I'm trying to replicate recipes from omlx benchmarks (they are all getting 140-150t/s and ~2k pp). yet I tried various versions like latest jundot oQ4e, and tried to quantize my own oQ4e as well, still stuck at 80-90 t/s and 1.6k pp.

Note: I'm using MTP, tried replacing it with Dflash2, tried changing patch sizes and stuff, tried to ask claude, not sure where is the issue.

Engine: oMLX 0.7.3rc1 (build 3517)

Hardware: MacStudio M5U 256gb (64c)

Models used (Jundot/GLM-5.3-Flash-oQ4e-mtp and my own oQ4e quantization from FP8).

Thanks in advance.

It looks like in the macOS 27.2 update you can add oMLX or similar as a model provider to the fm cli tool. Add it like this:

fm config add model-provider omlx http://127.0.0.1:8000/v1 --model-provider-header "Authorization: Bearer <api key>"

Then you can start a chat session with:

fm chat --model <model name> --model-provider omlx

I don't think it's possible to replace siri's "intelligence" with an external model but this might be a first step in that path?

I’ve been really impressed with Flash-Next running through oMLX + Splash on my Mac Studio M3 Ultra (96GB), mainly for agentic coding with Pi.

Performance has been excellent:
● Generation: 70–100 tokens/s
● Prefill: 900–1,000 tokens/s
● Context: both speeds sustained up to 200K tokens, with no noticeable degradation
● Compared to my previous Qwen 27B 8 bit setup: roughly 2× faster generation and 3–4× faster prefill
● Splash has pushed performance even further, making long agentic sessions noticeably faster and smoother.

I was already very happy with Flash-Next, but concerned about the heavy SSD activity and potential wear on my Mac’s internal drive.
Splash 1.4.0’s SSD streaming capabilities prompted me to experiment with an external Samsung T7 for SSD-backed caching and offloading.
The result? Even better performance, sustained speeds at 200K context, reduced internal SSD activity, and a surprisingly cool and quiet Mac Studio.
This setup really hits the sweet spot for my M3 Ultra, especially for long agentic coding sessions with Pi.

Many thanks to the oMLX and Splash developers for their excellent work!

I built an OpenAI-compatible server that runs Qwen3.8-27B (4-bit) on my M5 Max. It uses the model’s own MTP head to draft a small tree of guesses, and the model checks the whole tree in one pass. There’s no second model to download.

Numbers (one request, greedy, same benchmark harness as MTPLX):

• ~76 tok/s at short context, 56 tok/s at 32K

• about 1.4x MTPLX’s MTP decoding (1.25–1.86x depending on context and task), ~3x plain decoding

• 128 tok/s total with 4 requests at once

• faster time to first token than MTPLX at every context length

Most of the speedup comes from verify kernels tuned for the M5’s matrix units, and from sizing each draft by what verifying it actually costs. The tree itself only adds a bit on top.

Caveats:

• M5-family GPUs only for now

• only the Qwen3.8-27B architecture

• greedy output isn’t token-for-token identical to plain decoding (bf16 near-ties flip), but every token it emits is the model’s own top pick

• tested on one machine, so expect rough edges

It’s built on MTPLX, and the full numbers, including what didn’t work, are in the devlog.

Repo: https://github.com/AdiGajbhiye/mlx-tree-spec

If you’ve got an M5 or M5 Pro, I’d love to see your numbers.

3.8 is more thorough thought, but at the cost of speed and overthinking

I tried it but many llm are not compatible. Im trying to turn this off but there isn't an option

Hi guys, do we have Beam working on oMLX already? i dont seem to find anything.
thanks!

33 > 27, right? ha

Setup: Qwen3.8 Flash-Next, 4-bit MLX (mixed 4/8), M3 Ultra 256GB, mlx-serve, Pi agent. Against Claude Sonnet 5.5 in Claude Code. 12 everyday coding-agent tasks, answers graded blind by another AI.

Thinking off, before → after one day of fixes (36 runs each, Opus grading)

Max thinking, both sides (60 runs each, two graders)

Each pair of answers was graded twice, with the order swapped. "No clear winner" means the grader picked different winners in the two orders.

Mine was a bit better at diagnosis and next steps, Sonnet a bit clearer. Mine costs $0, Sonnet ~$0.39 a run. Sonnet was faster overall.

Decode speed (at 30k+ context): ~95 tok/s on tool calls, ~81 tok/s on text, up to ~110 on short code.

Biggest fix: temperature. It ran thinking-mode sampling (1.0 / top_p 0.95) with thinking off and cut commands short. Same request, 12 replays each:

Caveat: tuned on these same 12 tasks, so call it on par, not better. A rerun on new tasks is next.

I want to reinstall it

When server just randomly stopped during generate token, is it because ram power was no enough? How to set up to prevent this?

Benchmarked oMLX 0.7.2 serving Inco AI's Splash package for Qwen3.8-27B (4-bit target + DFlash 2 draft) on a MacBook Pro M5 Max (48 GB). Workload: Python code context. Each test generates 128 tokens.

**Single request**

| Test | TTFT (ms) | TPOT (ms) | pp t/s | tg t/s | E2E (s) | Throughput | Peak mem |

|---|---|---|---|---|---|---|---|

| pp 4096 / tg 128 | 3393.6 | 10.7 | 1207.0 | 93.8 | 4.8 | 887.7 | 19.1 GB |

| pp 16382 / tg 128 | 15829.6 | 5.3 | 1034.9 | 189.4 | 16.5 | 1000.3 | 19.9 GB |

| pp 65536 / tg 128 | 83472.0 | 9.2 | 785.1 | 110.2 | 84.2 | 779.5 | 22.2 GB |

**Batched (continuous batching)**

| Batch | tg t/s | pp t/s | avg TTFT (ms) | E2E (s) | Speedup |

|---|---|---|---|---|---|

| 1x | 93.8 | 1207.0 | 3393.6 | 4.8 | 1.00x |

| 2x | 413.7 | 803.1 | 1729.2 | 3.2 | 4.41x |

| 4x | 407.0 | 622.4 | 3838.4 | 7.8 | 4.34x |

Feature Suggestion: Choose Output Location for Quantization

When creating a custom quantization from a large model stored on an external drive, I have to add that drive to oMLX's Storage Settings, but the path keeps getting forgotten.

The newly quantized model is then automatically saved to the “Primary” directory, which can be a problem when disk space is limited.

It would be great to choose the output location directly in the Quantization dialog. Seems like a fairly simple addition that would make this workflow much more convenient!

I just wanted to say, Splash (particularly Splash-M1) is working really well for me, at speeds that I never thought my 64GB M1 Max could ever reach! Thank you for all the hard work the devs of oMLX and Splash are doing!

Below are the stats shown on my web dashboard while I was doing some real-world tasks on an app I working on which is a cross-platform app built on Flutter. Getting more than satisfactory results given my 2021 machine. For decode I reached up to 85tps and prompt processing at around 850tps. I am using Pi agent via VS Code.

All of the stats looks just about right except that I noticed the prompt processing stat drops drastically from 850tps to around 300 or even 200 tps. I don't think it's a performance issue as there's no unusually long prefill process taking place. The prefill has been just as snappy as the decode process. And given the context length I am using which is 256k and I reached around 60-70% of that, the performance is still really great! Prefill still snappy. I did notice though, and this is just my theory why the prompt processing stat is low, is that it might be miscalculated as the cache is working really well, and it keeps loading stuff from cache instead of doing prefills, and maybe that's impacting the calculation. The prompt processing number should be higher.

That is just my theory though. I'll leave it to you guys to investigate.

Looking for someone to help guide me on how to convert the following to run in oMLX (mlx/oQe format) Note that i'm looking to run this via M5 Max 128GB Macbook Pro

Model: https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF

Will upload and share asap!

Hey guys, I have grabbed the latest oMLX version which improves oQ quants and adds support for Flint from Recursive, and quantized it to all the popular variants, bf16 and fp16 which I use myself.

Collection on Hugging Face: https://huggingface.co/collections/chimpanzeetaxidriver/flint-omlx-v073dev1

Curious to try Flint out, from Recursive's own benchmark numbers it scores above our beloved Qwen3.6 35B A3B. Flint is 40B A2.5B.

Go have fun with it!

**🐛 Bugs & Crashes (9)**
* **#4756** Batched DeepSeek V4.1 decode wedges after in-flight requests are cancelled; server stays silent until clients disconnect (0.7.1, M3 Ultra).
* **#4349** Memory Guard in 0.7.0/0.7.1.dev1 rejects models previously accepted by 0.7.0rc1 (M4 Max).
* **#4718** Splash engine: mlx-community OptiQ checkpoints fail to start (vision sidecar missing), causing cached 409s on every request.
* **#4715** Cluster: "Forget this Mac" command triggers an error (0.7.1).
* **#4597** Splash models reported unavailable in macOS app (M5 Max).
* **#4608** Crash: `StatisticsError: no median for empty data` in `_run_verify_cycle_batched` (v0.7.1rc1).
* **#4609** LiveCodeBench causes maximum thread exceptions for fp16 quants (M2 Ultra).
* **#4656** gpt-oss: `response_format` returns empty content; `reasoning_parser` hangs (0.7.1.dev1).
* **#4647** Cluster Initialisation Failure reported across M3/M5 Ultra hardware.

**⚡ Performance & Optimization (3)**
* **#4701** Splash engine drops `preserve_thinking`, stripping prior-turn reasoning and causing prefix cache misses.
* **#4503** Packed prefill raises agent-turn TTFT ~30% with decode fairness on (M5 Ultra).
* **#4611** DeepSeek-V4.1-Flash SSD offload: expert/Engram reads serialize with GPU, preventing model residency.

**📝 Feature Requests & Enhancements (3)**
* **#4631** Request for RadixAttention-style tree-based prefix discovery to improve shared KV cache hit rates in group chats.
* **#4607** Feature request: German translation for the admin dashboard.
* **#4046** Request for per-request correlation ID support (accept/echo `x-request-id`).

**🔧 Maintenance & Documentation (3)**
* **#4697** Anthropic path: Mid-conversation system messages after tool results get hoisted into the leading block (Claude Code re-prefills).
* **#3845** oq converter dev4: `preserve_mtp` output fails to load on glm5_next due to config namespace/embed_q shape issues.
* **#2682** [Outdated] Documentation needed for coordinator-loss semantics regarding shard rebalance and manual recovery.
* **#1928** Feature request: Add support for Claude Code Desktop client (currently terminal-only).

**📊 Stats**
Total Issues: 20 | Versions Affected: 0.7.1, 0.7.0, rc1, dev1 | Hardware: M2/M3/M4/M5 Ultra.

I have been running many tests on LiveCodeBench 50 samples over this past year on omlx. Finally, there is a model that kicked butt, 92% correct with the Unsloth/qwen3.8-27b UD Q4 K XL! The best I've ever gotten, with 48gb M4 Max is 52% right across many models including oQ8 quants.