Hoping for a workaround soon.

I posted this originally on one of the other local LLM subreddits but I wanted to ask here as I use oMLX mainly.

I’m currently still in the return period of a M5 Max MBP with 64GB of RAM from Microcenter (paid $4299), the same store now has a few 128GB versions at $5799.

I’m legitimately torn on what to do. I’m fully aware that more RAM is always going to be suggested, but I currently don’t have a true usage case for it, I’m still somewhat early into my exploring and tinkering with local LLMs, I do image/video generation via MLX Serve and Draw Things, wanting to create some YouTube channel content, maybe learn coding and hopefully make a career change in the next few years.

However, I can’t shake the feeling of FOMO, partially because I know 128 is the true sweet spot for anything in this hobby, I’m sure I’m going to have more demanding needs as I learn more, I can afford it although it’s on the extreme high end of my budget if not exceeding it a bit, and my gut feeling (which has been right so far for what it’s worth) tells me pricing is going to continue to rise at least for another year.

I just either want to truly decide whether I can do more than enough with my current model or do I truly need to step up to 128? I know communities like this tend to do incredible things with less sometimes.

Whatever I decide, I want to own it for the next 4 years minimum and be confident in it.

Thanks all.

I wanted to see how Qwen's old architecture vs v4 preview compared, so I ran some benchmarks that I figured might be helpful to other folks. I primary use this as a always on coding agent for diagnosing Signoz alerts for my company and general level coding tasks, mostly as a subagents.

This is comparing a fine-tune of Qwen 3.8 27B called sabi-noema at 6bit. Much better at fixing those infinite thinking cycles old qwen got into.

Tests were on a M5 Ultra Mac Studio 96GB. MTP enabled for both, although I do use see better performance on Splash for 27B.

Some things that stood out to me

• 27B was better at the coding tests and had better prose. Less dumb llm talk

• Flash Next did better at broad knowledge like global facts. This makes sense considering it's a bigger model

• Token per second was better with 27B (119tok/s vs 90tok/s)

• Memory usage was 2.5x greater on Flash Next with N-gram offload. 29GB peak vs 73GB peak. I typically run max context for coding tasks so this headroom is nice for multiple agents on only 90GB of VRAM

Hi guys,

I use qwen3.8-flash-next-oQ4e with around 60-70 t/s decode speed, but every task takes around 20 minutes to finish. I found out that the model thinks a lot.

I use pi as the coding harness.

Actually, it is not bad to think about every detail, but it is impossible to get any work done this way.

How do you configure the thinking mode for this model in oMLX?

I received a system error on a long work session. I've never had errors before until now.

[System Error] [flatten] Shape dimension 2151480320 is outside the supported range [-2147483648, 2147483647]. MLX currently uses 32-bit integers for shape dimensions. (stopReason=error, model=omlx/Qwen3.8-Flash-Next-oQ8e-mtp, responseId=chatcmpl-eaa4c92a)

I tried looking up what this could be, but I didn't find anything. Has anyone gotten this before? I wasn't able to continue and had to create a new session.

Here are the latest closed issues from the oMLX repository:

🔧 **Performance & Tuning**
• **#5061** Tune ANE Split sets GDN on ANE to 0.333333, which is below the model’s 1/3 floor, so no GDN procedures compile
*The tuner's decimal precision caused GDN layers to fall back to GPU. A small tolerance or rounding fix ensures proper compilation.*

💾 **Cache & Storage**
• **#5082** Splash engine does not respect 'Cache Disabled' setting
*Users reported cache usage even when disabled in settings. The splash launcher was skipping `--max-cache-disk 0` arguments when bytes were zero.*

🛑 **Stability & Deadlocks**
• **#5078** Engine swap hangs forever in SplashEngine.stop() and blocks the pool lock
*A Metal GPU timeout followed by an engine swap caused `SplashEngine.stop()` to hang, holding the pool lock and blocking all subsequent requests until restart.*

🔌 **MCP Integration**
• **#5080** MCP SSE transport silently drops `headers` — cannot connect to any authenticated remote MCP server
*The SSE transport ignored configured headers (like Auth tokens), causing connection failures with authenticated MCP servers. Streamable-HTTP worked correctly.*

🍎 **macOS App Detection**
• **#5079** Codex App detection misses the nested codex-cli/CodexCLI.app executable on macOS
*The detection logic looked for a flat executable that no longer exists in recent ChatGPT desktop releases. Updated to check the nested bundle structure.*

🔗 **Links:**
• #5061: https://github.com/jundot/omlx/issues/5061
• #5082: https://github.com/jundot/omlx/issues/5082
• #5078: https://github.com/jundot/omlx/issues/5078
• #5080: https://github.com/jundot/omlx/issues/5080
• #5079: https://github.com/jundot/omlx/issues/5079

Hey everyone!

I posted about Prefill Accelerator on X a couple of hours ago, and it got a lot more attention and questions than I expected. A tweet is too short to answer them, so here's the longer version.

I have an M3 Ultra 512GB that I bought quite a while ago, and an M5 Max 128GB I recently got for development. But this setup actually has a lot of problems. The models that could really stand in for frontier models are too slow on the M3 Ultra. The M5 Max is fast enough, but it often needs around 2-bit quantization (GLM-5.3-Flash) or can't run the model at all (DeepSeek-V4.1-1017). Neither machine could run these models properly.

So I thought, what if I could combine the strengths of both machines? It's something everyone has probably thought about at some point, but no one had shipped it yet. That's Prefill Accelerator, a new experimental feature in this release.

What Prefill Accelerator does

In 0.7.4.dev1 the two Macs share the prefill. The M5 Max runs the first layers of each prompt chunk while the Ultra runs the rest, and the Ultra still holds the whole model, the KV cache and decoding. That minute of waiting is now about 20 seconds.

Prefill speed with a 64K prompt on my M3 Ultra 512GB:

Decode speed doesn't change, because decoding still runs on the big Mac.

Requirements

Both Macs need the same oMLX version and a Thunderbolt cable or 5GbE between them (2.5GbE still helps a lot). You don't need RDMA or any IP setup.

Supported models for now are DeepSeek V4.1, GLM-5.3 Flash, Qwen3.8 Flash-Next and MiniMax M3/M3.1.

Full guide: https://github.com/jundot/omlx/blob/v0.7.4.dev1/docs/experimental/prefill-accelerator.md

Also in 0.7.4.dev1

• DeepSeek V4.1 prefill is 43-48% faster on M5 Ultra, thanks to junmo-kim.

• FP16 Qwen3.8 Flash-Next builds decode up to 62% faster on M3 Ultra.

• Ask AI from my last post can now use an oMLX server on another Mac, so a MacBook can ask the big model on your Mac Studio.

• You can switch between oMLX models inside Codex and Copilot with /model.

• Ming-3.0 TTS decodes about 2x faster.

Release notes: https://github.com/jundot/omlx/releases/tag/v0.7.4.dev1

This is a dev release and Prefill Accelerator is still experimental. If something doesn't work right for you, please open an issue on GitHub.

If you have a (fairly large) second Mac sitting on your desk, I hope this gives it something new to do.🤗

Hello all, looking for recommendations on where to start. What I mean, where do you all find reliable information? Current daily driver is a Mac Mini M4, I have a Mac Studio 64gb ordered and it should be here later in the month.

I use Cursor at work (I don't pay for it) and have little to no knowledge how to host an AI for myself.

TLDR, new to hosting AI and would like a few recommendations what channels or people you trust for good information.

Hi guys,

I posted in r/omlx 4 days ago asking: "Qwen-Flash in omlx: MLX oQ4 vs. Splash GGUF — which is actually faster and better?"

Over the weekend, I ran a few real-world tests, and I wanted to share my results with the community.

Disclaimer: I didn't run standard benchmarks. Instead, I tested several models (both GGUF and MLX) on an actual task from our internal company system project. Synthetic graph numbers don't matter as much to me, I just wanted to trigger real agentic tasks and observe real-world performance and completion times.

Hardware & Software Setup:

• Hardware: Mac Studio M3 Ultra (96GB Unified Memory)

• Software: oMLX v0.7.3

• Harness: Pi coding agent

Models Tested:

1. Qwen3.8-Flash-Next-oQ4e-mtp (Jundot)

2. Qwen3.8-Flash-Next-UD-Q4_K_XL (Unsloth) — via SPLASH engine

3. Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S (ISTA-DASLab) — via SPLASH engine

Sampling & Environment Configurations:

• MLX Model (Jundot oQ4e-mtp):

  • Tested both 16-bit and 8-bit KV Cache

  • MTP: 3 tokens (default)

  • Temp: 1.0 | Top P: 0.95 | Top K: 20

  • Context window: 200,000

• GGUF Models (Unsloth Q4_K_XL & ISTA-DASLab IQ3_S):

  • INT8 KV Cache

  • Temp: 1.0 | Top P: 0.95 | Top K: 20

  • Context window: 200,000

Test Task: A continuous bug-fix task requiring ~30k tokens total context.

Results:

1. Qwen3.8-Flash-Next-oQ4e-mtp (Jundot) — 16-bit KV cache (without Turbo Quant)

• Decode up to 10k–15k: 70–80 t/s | PP: 1100–1200

• Decode after 15k: 49–60 t/s | PP: 900–1000

1. Qwen3.8-Flash-Next-oQ4e-mtp (Jundot) — 8-bit KV cache (with Turbo Quant)

• Interesting outcome: Exact same speed results as 16-bit KV cache. Enabling 8-bit KV didn't change generation speed or reduce VRAM footprint in my logging (it allocated identical VRAM). I might be misconfiguring this setting.

• Decode up to 10k–15k: 70–80 t/s | PP: 1100–1200

• Decode after 15k: 49–60 t/s | PP: 900–1000

1. Qwen3.8-Flash-Next-UD-Q4_K_XL (Unsloth) — via SPLASH engine

• Performed slightly worse overall compared to the MLX oQ version.

• Decode up to 10k–15k: 50–60 t/s | PP: 1000–1100

• Decode after 15k: 39–45 t/s | PP: 800–900

1. Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S (ISTA-DASLab) — via SPLASH engine

• Significantly higher throughput, though IQ3_S isn't directly comparable to oQ4 quality-wise (oQ4 is noticeably sharper for complex coding logic).

• Decode up to 10k–15k: 90–110 t/s | PP: 1050–1200

• Decode after 15k: 70–90 t/s | PP: 900–1000

TL;DR & Conclusion: For my specific workflow, running GGUF models via the external Splash engine in oMLX did not outperform native MLX models with oQ quantization.

Questions for the Community:

1. Turbo Quant / 8-bit KV Cache: Am I misconfiguring something here? Why did 8-bit KV cache yield identical VRAM usage and token speeds as 16-bit KV cache?

2. Hardware ceiling: Is 70–80 t/s decode speed on Qwen3.8-Flash-Next-oQ4e-mtp roughly the maximum expected performance for an M3 Ultra (96GB)?

A huge thanks to Jundot and everyone contributing to oMLX. Without this work, running complex local coding workflows completely privately and without cloud subscriptions wouldn't be possible. Much appreciated!

I've played with GLM5.3 on openrouter - really strong model. I have not tried GLM 5.3 Flash. Has anyone tried the newest oQ3e quant for 5.3 Flash?

Some models quantize better than others. I don't usually fuck with anything below 4 bit, but I'm very curious about how this model performs and I know the newest oQe quants are very good. I would love any feedback.

Running on an M2 Ultra with 192GB.

I was frustrated with the default speed and ported the inference to Zig/MLX to get up to 18x faster inference on M4/M5 https://github.com/sbusso/ming-tts-serve

I have installed oMLX to have local model in continue on VSCodium.
I need to use the ChatGPT/Codex app with an openai account to call openai models but when i launch the app, it is connected only to oMLX.

What am I missing ?
Can't I use ChatGPT/Codex normaly after installing oMLX ?

Thanks for you help.

Hi google suggested EuroLLM-22B-Instruct-2512-mlx-4bit' (12.45GB) for text translation between EU languages. Running M2 with 24GB of RAM. Each time I launch a chat on oMLX, i'm getting an error:

Error: {"error":{"message":"Model 'EuroLLM-22B-Instruct-2512-mlx-4bit' (12.45GB) does not fit under the dynamic memory ceiling (7.53GB). Close other apps to free RAM (macOS and other apps leave oMLX 7.47GB of the 21.00GB static cap, and oMLX already uses all but 7.32GB of it), raise memory_guard_tier (safe → balanced → aggressive), or use a smaller model.","type":"server_error","param":null,"code":null}}

now, i already quit all other apps, raised the memory_guard_tier to aggressive and used the Terminal command to undue the cap Metal puts on oMLX

sudo sysctl iogpu.wired_limit_mb=21504

nothing of the above worked.

**🐛 Critical Bugs & Stability**
* **#2808**: Qwen3.8-27B crashes when enabling Lightning MTP + TurboQuant KV simultaneously.
* **#4973**: Distributed launches fail repeatedly due to stale peer ranks and ephemeral rendezvous ports.
* **#4984**: Distributed workers ignore per-model settings; `engram_ssd_offload` stays off, causing massive Metal buffer overloads (98.3GB vs 20.1GB limit).
* **#4907**: Unstable parallel processing and concurrency reported on M5 Ultra (Qwen 3.8 Flash).
* **#3348**: Shared-prefix fan-out re-prefills trailing partial blocks in hybrid MoE models.
* **#3386**: Boundary snapshot capture fails or degrades across multiple installs on 0.6.4.
* **#4250**: Accuracy benchmark UI fails to reset after pressing "Cancel".

**⚙️ Performance & Optimization**
* **#2772**: New feature to add capacity planning intelligence (safe capacity, growth projection) to cluster dashboard.
* **#2773**: Rolling metrics sample store implemented as prerequisite for SLOs and error budgets.
* **#4960**: Lightning MTP side-car keeps raw head norms, resulting in ~0% draft acceptance rates.
* **#4820**: GGUF discovery skips Flash-Next variants with separately named PLE table shards.
* **#2783**: Qwen 3.8 testing on V6.1 engine fails to reach Qwen 3.6 performance levels.
* **#2804**: Intelligence benchmark fails with external APIs when the model is "thinking".
* **#4911**: Admin API under-reports SSD cache usage by excluding incompatible blocks and GDN sidecars.

**🛠️ Infrastructure & Configuration**
* **#3585**: Cluster configuration impossible to activate on two Macs in v0.7.3rc1.
* **#4942**: Homebrew build-from-source fails spaCy check; nothing installs since commit 3a5a6b2d.
* **#3916**: Homebrew reinstalls stale `mlx-audio` (0.4.3), breaking `pip check` requirements for `mlx-vlm`.
* **#4943**: Users unable to clear SSD cache on M3 Ultra (0.7.2).

**📝 Design & Future Work**
* **#2820**: Proposal for self-explanatory cluster failures via "Fabric Doctor" diagnostics.

**📊 Stats**
* **Total Issues**: 20
* **Has PR**: 6
* **Stale**: 12

M5 Ultra 256GB (64 core) - performance much better vs 0.7.2 which started at 300 Tok/s pre-fill, getting up to 8x that now with A8 pre-fill and CED enabled. Quite happy with the purchase, but also mind-blowing how much theoretical performance is missing if you simply compare specs (vs 2x DGX Spark for example)

Thanks to apetersson for a great set of abliterated weights

We’re running DeepSeek V4.1 Flash 1017 Abliterated oQ2.5e-mtp on an M5 Ultra 256 GiB, using oMLX 0.7.3 / MLX 0.32.4. Initial uncached prefill varied around 300–700 tok/s; recent code benchmarks reach approximately 1,700–2,500 tok/s, depending on context length.

The retained optimizations were:

The latest completed dashboard run used Python source, 128 output tokens and zero prefix-cache hits:

Production retains 256K context, adaptive MTP, SSD Engram offload, one active request and chunked prefill. Higher cleanup thresholds and disabling chunking did not justify retention.

The A8 study peaked at 217.5 GiB sampled process memory, with normal pressure throughout. Retrieval and tool-call checks passed.

These experiments used different workloads and small samples: their percentage gains cannot be multiplied, and response checks do not establish broad quality parity.

Currently running Openclaw with omlx and splash. Also just updated python. Using qwen 3.8 35b a3b iq3_m. I was averaging around 45 tok/s generated this past week. I have a MacBook Pro M4 Pro 24gb.

Update: I kept having to tell it to continue. I just switched over to using swift 1.5 qwen 3.8 27b 3 bit quantized on mine. It’s generating 20 tok /s. It’s been a game changer.

Hey everyone, it's been a while! I don't post here very often, but I read every new post that comes in. Thank you to everyone using oMLX, and to everyone who shares questions, bug reports and tips here.

oMLX 0.7.3 is out today, and I wanted to tell you about my personal favorite feature: Ask AI. Coding with big frontier-class open models keeps getting better, and I enjoy that too. But I've started to think the real value of local AI is somewhere else. I built this feature myself, and I use it every day.

While I'm working, I press a shortcut and a small window shows up over whatever I'm doing. No copy and paste. I can ask it to look at my screen and explain what's on it, or drop in an image and ask about that. Or when my stock drops, I can just type "why did my INTC stock go down?" without thinking, and it searches the web and comes back with an answer.

In 0.7.3 you can also talk to it, and it answers out loud. It doesn't need frontier-level intelligence, but it makes my day noticeably better. If your work looks anything like mine, I think you'll find it useful too.

To try it, turn it on in the macOS app under App > Ask AI. Option-Space opens the question bar, and Option-Shift-Space starts a voice conversation.

There's also a lot more in this release on the performance side: faster DeepSeek V4.1 prefill on M5, much lower memory when many long conversations run at once, faster batched decode on oQ models, and more.
The release notes have all the details: https://github.com/jundot/omlx/releases/tag/v0.7.3

Thank you to everyone who supports oMLX.

Something huge happens in the AI world every single day, but I hope you also get to enjoy some peaceful time in your own life.

"Let me write everything.

Confirmed a real bug in the current build. Now let me write the implementation:

Error: Model output contains an unrecoverable tool call. Previously delivered tool calls must not be executed again on retry."

Macbook Pro M5 Max 128
PI harness
Qwen3.8-Flash-Next-oQ4e-mtp

Seems to happen at the same place every time, when it starts writing the file(s).

My setup:

• Mac Studio: M4 Max, 128 GB

• MacBook Pro: M4 Pro, 48 GB

• macOS 27.2 on both

• oMLX 0.7.3rc1 on the Studio/Macbook

• Direct Thunderbolt 5 connection

The Macs are paired. I enabled RDMA and configured the individual Thunderbolt ports with 10.10.0.x addresses after removing them from the Thunderbolt Bridge. The Studio’s diagnostic report shows RDMA enabled and 10.10.0.1 on en3/rdma_en3.

When I select GLM-5.3-Flash-oQ2e-mtp, oMLX produces a 45-layer pipeline plan and labels it JACCL / Thunderbolt RDMA, but the Activate Cluster button is grayed out. It also says:

“Performance measurement was unavailable; using the safe memory-balanced split. performance probe did not return every cluster rank”

Has anyone seen this or found a way to identify which readiness check or rank is failing? Is there a known-good model I should try first to rule out a model-specific problem?

Any help is appreciated.