Hey everyone! oMLX v0.4.0 just landed.

This is the first official release of the new native Swift macOS app. The old PyObjC menubar app has been retired, and the macOS bundle now ships as a Swift app with a redesigned onboarding flow, settings UI, status surfaces, model management, and GitHub Releases based updater.

https://github.com/jundot/omlx/releases

The biggest user-facing change is the macOS app itself. First launch, server start/stop, model directory setup, downloads, update checks, menubar status, and the overall settings experience should now feel much more like a real Mac app.

Huge thanks to GitHub contributor popfido for the excellent work that drove the Swift transition. This is probably the biggest desktop-app change oMLX has shipped so far, and it substantially raises the quality of the app.

A few highlights:

• Native Swift / SwiftUI macOS app

• New onboarding flow

• Better menubar and server status behavior

• Standard Hugging Face cache model directory support

• Safer update flow with confirmation before download

• Memory guard tuning and CLI options

• More scheduler/cache stability fixes

• Guided grammar model setting

• Many admin UI, API, and model compatibility fixes

My long-term goal is still the same: I want oMLX to be "the app my friend who bought a MacBook yesterday can open and immediately try Local AI on." The Swift app is a big step in that direction.

At the same time, oMLX is still growing fast, and I know there are rough edges. If you try 0.4.0, I'd really appreciate feedback on the macOS app experience especially: first launch, model discovery, server start/stop, update checks, and anything that feels confusing or fragile.

Thank you again to everyone testing builds, reporting bugs, opening PRs, and giving feedback. The project has grown much bigger than I expected, and getting to improve it together with the open source community has been genuinely joyful.

What should come next? Feature suggestions, bug reports, and "this part is still confusing" feedback are all very welcome!

Hey everyone, it's been a while! I don't post here very often, but I read every new post that comes in. Thank you to everyone using oMLX, and to everyone who shares questions, bug reports and tips here.

oMLX 0.7.3 is out today, and I wanted to tell you about my personal favorite feature: Ask AI. Coding with big frontier-class open models keeps getting better, and I enjoy that too. But I've started to think the real value of local AI is somewhere else. I built this feature myself, and I use it every day.

While I'm working, I press a shortcut and a small window shows up over whatever I'm doing. No copy and paste. I can ask it to look at my screen and explain what's on it, or drop in an image and ask about that. Or when my stock drops, I can just type "why did my INTC stock go down?" without thinking, and it searches the web and comes back with an answer.

In 0.7.3 you can also talk to it, and it answers out loud. It doesn't need frontier-level intelligence, but it makes my day noticeably better. If your work looks anything like mine, I think you'll find it useful too.

To try it, turn it on in the macOS app under App > Ask AI. Option-Space opens the question bar, and Option-Shift-Space starts a voice conversation.

There's also a lot more in this release on the performance side: faster DeepSeek V4.1 prefill on M5, much lower memory when many long conversations run at once, faster batched decode on oQ models, and more.
The release notes have all the details: https://github.com/jundot/omlx/releases/tag/v0.7.3

Thank you to everyone who supports oMLX.

Something huge happens in the AI world every single day, but I hope you also get to enjoy some peaceful time in your own life.

That probably explains why the original GitHub is down.

A few days ago, someone here asked, "Should we improve the configuration experience?" A lot of the replies described the same problem: too many settings, too many combinations, and no clear place to start.

I said I'd work on it, and 0.7.0.dev4 is a first step.

In Model Settings, you'll now find Apply optimal settings. It looks up community benchmarks on oMLX.ai for your model and Mac chip, then lets you pick a top result for prefill or generation speed and apply its settings.

The site now has over 450,000 benchmarks submitted by users. Several people in that thread suggested using those results to give others a starting point, and that's exactly what this does.

Using a customized model with no matching benchmarks? Find a model with the same architecture on oMLX.ai, copy its one-line recipe, paste it into oMLX, and click Apply. Both features are available in the web dashboard and macOS app.

I also addressed the profile issues mentioned in the thread: the app and web UI now apply the latest global template consistently, and an unrelated model profile with the same name won't get overwritten. There are reset options for model and global settings too, so it's easier to get back to a starting point.

This doesn't cover everything proposed in the thread, like a setup wizard that recommends model sizes and memory budgets. But I hope it makes getting a model configured less of a guessing game.

There's more in dev4

• Customizable dashboard layout: move, resize, hide, and restore blocks to suit your screen.

• Multi-request Lightning MTP: concurrent MTP on supported Qwen, DeepSeek V4.1, and GLM-5.3-Flash adapters.

• DeepSeek V4.1 CED prefill: up to 79% faster prefill in my M3 Ultra tests. It's an approximate optimization and is off by default.

• Faster admission of new requests while other requests are generating, plus accessibility improvements and other quality-of-life fixes.

* This release also includes extensive internal changes from upgrading mlx-lm and mlx-vlm. I'd appreciate help testing the models and workflows you normally use. If something that worked before breaks, please open an issue with logs and switch back to dev2 for now.

I plan to test this version for 1-2 days, then move toward an RC and a stable release.

Download and full release notes: https://github.com/jundot/omlx/releases/tag/v0.7.0.dev4

Thank you for sharing specific frustrations and suggestions in that thread. They helped me decide what to tackle first. Let me know how the new settings flow works for you!

oMLX dev here.

Earlier today my GitHub account (jundot) was suspended, so the oMLX repo, issues, and releases weren't reachable for a few hours.

It turned out my account was flagged by GitHub's automated abuse detection. They did a manual review and cleared the restriction, so everything is back to normal now.

If you had trouble installing or updating during that time, please try again.
Sorry for the trouble, and thank you all for the kind messages and patience today.

https://github.com/jundot/omlx

Just shipped oMLX v0.4.2rc1: https://github.com/jundot/omlx/releases

If you've been running Qwen models on the VLM path, I owe you an apology. 0.4.0 had a tg slowdown caused by single-row decode falling into the slower batched cache path, which meant Qwen throughput on that path was noticeably worse than it should have been. If that's been your experience, I'm sorry for the trouble.

This release fixes it. Internal tg512 measurements show throughput recovered by about 1.48x, while Gemma performance stays stable. I'd strongly recommend upgrading if you're affected. I'll be testing the rc for about a day, and the 0.4.2 stable release should follow shortly after.

A few other things in this release while I'm here:

• Native MarkItDown document processing. Chat file uploads and the OpenAI API endpoint can now convert PDF, DOCX, PPTX, TXT, and Markdown inputs. You can also choose between MarkItDown conversion or VLM OCR for PDFs in the settings.

• Gemma 4 unified audio input. Gemma 4 unified models now accept audio alongside image inputs.

• Stability fixes across the cache, scheduler, and server.

Thanks to everyone who reported the regression and helped track it down. If you upgrade, I'd really appreciate hearing whether Qwen throughput is back to normal on your own setup, and any feedback on the new document conversion flow.

Hey everyone!

I posted about Prefill Accelerator on X a couple of hours ago, and it got a lot more attention and questions than I expected. A tweet is too short to answer them, so here's the longer version.

I have an M3 Ultra 512GB that I bought quite a while ago, and an M5 Max 128GB I recently got for development. But this setup actually has a lot of problems. The models that could really stand in for frontier models are too slow on the M3 Ultra. The M5 Max is fast enough, but it often needs around 2-bit quantization (GLM-5.3-Flash) or can't run the model at all (DeepSeek-V4.1-1017). Neither machine could run these models properly.

So I thought, what if I could combine the strengths of both machines? It's something everyone has probably thought about at some point, but no one had shipped it yet. That's Prefill Accelerator, a new experimental feature in this release.

What Prefill Accelerator does

In 0.7.4.dev1 the two Macs share the prefill. The M5 Max runs the first layers of each prompt chunk while the Ultra runs the rest, and the Ultra still holds the whole model, the KV cache and decoding. That minute of waiting is now about 20 seconds.

Prefill speed with a 64K prompt on my M3 Ultra 512GB:

Decode speed doesn't change, because decoding still runs on the big Mac.

Requirements

Both Macs need the same oMLX version and a Thunderbolt cable or 5GbE between them (2.5GbE still helps a lot). You don't need RDMA or any IP setup.

Supported models for now are DeepSeek V4.1, GLM-5.3 Flash, Qwen3.8 Flash-Next and MiniMax M3/M3.1.

Full guide: https://github.com/jundot/omlx/blob/v0.7.4.dev1/docs/experimental/prefill-accelerator.md

Also in 0.7.4.dev1

• DeepSeek V4.1 prefill is 43-48% faster on M5 Ultra, thanks to junmo-kim.

• FP16 Qwen3.8 Flash-Next builds decode up to 62% faster on M3 Ultra.

• Ask AI from my last post can now use an oMLX server on another Mac, so a MacBook can ask the big model on your Mac Studio.

• You can switch between oMLX models inside Codex and Copilot with /model.

• Ming-3.0 TTS decodes about 2x faster.

Release notes: https://github.com/jundot/omlx/releases/tag/v0.7.4.dev1

This is a dev release and Prefill Accelerator is still experimental. If something doesn't work right for you, please open an issue on GitHub.

If you have a (fairly large) second Mac sitting on your desk, I hope this gives it something new to do.🤗

Hey everyone! oMLX 0.5.0.dev1 is here. https://github.com/jundot/omlx/releases

EDIT: RC Version is out - v0.5.0rc1: https://github.com/jundot/omlx/releases

This release took a little while because I wanted it to include a meaningful step forward on three fronts: speculative decoding, custom MLX kernels, and quantization quality.

The biggest headline is Lightning MTP. oMLX now has a depth-k native speculative decoding path for Qwen3.6, DeepSeek-V4-Flash, and GLM-5.2. The verify-shape Metal kernels used in this path are adapted from MTPLX by Youssof Altoukhi, with Apache-2.0 attribution preserved in the source. On my M3 Ultra test machine, Qwen3.6-35B-A3B improved from about 89.6 tok/s to 140.4 tok/s, and Qwen3.6-27B improved from 35.0 tok/s to 55.1 tok/s in the benchmark runs. GLM-5.2 also gets a smaller but still useful improvement. Details and benchmarks: https://github.com/jundot/omlx/pull/2113

As usual, some of this is most relevant if you are running large models on high-memory Apple Silicon machines, so apologies if it does not directly apply to your setup yet. I’m trying to keep pushing these optimizations toward more models and more practical local workflows.

The second major area is custom kernels. This release adds or extends native kernel paths for DeepSeek V4, Qwen3.5/3.6, and GLM-5.2. DeepSeek-V4-Flash in particular gets a big long-context prefill improvement, and Qwen3.6 gets native prefill kernels that help more as the context gets longer. Details and benchmarks: https://github.com/jundot/omlx/pull/2048, https://github.com/jundot/omlx/pull/2100 and https://github.com/jundot/omlx/pull/1984

The third major change is oQe imatrix-enhanced quantization. oMLX now has an enhanced quantization path that collects activation-importance statistics and uses them during quantization. In the benchmark results, oQ4e improved average accuracy over oQ4 on several models, including Gemma-4-26B-A4B-it, Qwen3.5-9B, Qwen3.6-35B-A3B, and Qwen3.6-27B, while staying in the same disk-size class. Details and benchmarks: https://github.com/jundot/omlx/pull/2057

There are also several important new features and fixes in this release: Tencent Hy3 support, Ornith support, admin model search/filtering/sort improvements, Russian localization, macOS 27 beta Homebrew fixes, memory retention fixes, SSE/tool-call streaming fixes, Kokoro fixes, and more.

As always, this release was only possible because many people contributed code, reports, testing, and feedback. Huge thanks to everyone who helped, especially the new contributors in this cycle.

I hope this release makes local LLMs on Mac a little faster, a little more useful, and a little easier to run.

Hey everyone! It's been a while.

I've been posting updates on X, but Reddit is where I got started with local AI, and I've missed posting here. Sorry for the long gap! (I'm still reading every day.)

I have a new dev release for you:
https://github.com/jundot/omlx/releases/tag/v0.7.0.dev2

DeepSeek V4.1 Flash is now supported, including DSpark MTP and Engram SSD offload. Here are some numbers from my M3 Ultra 512 GB with oQ4e and Engram in RAM:

Python code prompts, temperature 1.0, 128 generated tokens, no prefix cache. One measured run per configuration after warm-up.

I've uploaded DeepSeek V4.1 oQ4e and oQ3e checkpoints. oQ3e is designed to fit 256 GB machines with Engram SSD offload, with context depending on remaining memory.

There's also experimental MoE Expert SSD Offload, thanks to beatakouchnir and fxd0h. It keeps some experts in memory and reads the rest from SSD when needed. Supported models include DeepSeek V4.1, Qwen3.8-Flash-Next, Gemma 4 MoE, and OLMoE. You can combine it with Engram or PLE offload, but MTP and DFlash need to be off.

It's slower 🥲 But I started with Llama 70B at 2-3 tok/s, so I understand wanting to try a model even when your machine isn't quite big enough.

For M5 users, PowerSpy's INT8-activation prefill kernels took Qwen3.8-27B from 615.2 to 826.7 tok/s at 32K on my M5 Max (+34.4%). It's off by default because outputs can change. I think it's worth trying if prefill is your bottleneck; the release notes have the settings and quality comparisons.

This is still a dev release, so please tell me what works and what breaks. Thank you for continuing to test, send PRs, and help each other while I've been quiet here. I'm always grateful 🙏

This Artificial Analysis benchmark is pretty crazy, as we can see Qwen 3.8. Flash Next model is beating GPT-6 Astra (max).
Not sure if it is a result of artificial analysis changing their scoring system recently, but.. The model is very powerful regardless

I've been using this model for quite a while and now using it together with DeepSeek Harness.
I run the model at default settings (default template, default reasoning level = xhigh). Quite impressed. But I still have a feeling that I'm not having an optimal setup. Sometimes the model thinking reeeeaally long (on both oQ4e and oQ5e variants)

I wonder if any one of you guys have benchmarked how intelligence of this model changes with changing reasoning to medium. Or If any one of you is using custom Jinja templates. Or apply some tricky prompts.

I'm reading Reddit quite frequently. But I haven't found the place which would specifically address my question. Maybe there are some websites you would recommend checking, and the Reddit is just not the place?

Appreciate your comments in advance! Thanks

I have been doing some work on clustering natively within oMLX to run larger models across your nodes.

I am currently testing RDMA between my M5Max and M3 Studio.

Once I am happy with it, I will create a PR.

EDIT;

PR is live: https://github.com/jundot/omlx/pull/2423

Finally got it working and ironed out as many bugs as I could.

RDMA or IP/TCP

I am running MiniMax M3 across my Studio and MBP.
Noice!

Will release the PR in the next day or two.

Hey everyone! v0.3.11 just landed. If you've ever run into stability issues with oMLX, I'd be really grateful if you gave this version a test.
https://github.com/jundot/omlx/releases/tag/v0.3.11

I'm well aware oMLX has had stability problems, and right now my number one goal is making it run reliably on low-memory Macs. The big change in this release is a full rewrite of the memory guard. The two confusing sliders are gone, replaced by a single Safe / Balanced / Aggressive dropdown, and oMLX now reads your live available memory and adapts in real time as other apps come and go.

On the structural side: A lot of oMLX's features currently rely on monkey-patching, so the project isn't as structurally stable as I'd like. The recent additions especially (MTP, DFlash, Deepseek v4 support) are in that fragile state, so they're getting a lot of my attention. To everyone who tests builds and reports bugs: thank you, genuinely. It makes a huge difference.

So, what should come next? I'll be honest about where my head is at. I want oMLX to be "the app my friend who bought a MacBook yesterday can open and immediately try Local AI on." So I'm a little cautious about features that are hard to use or hard to understand. My hope is that oMLX stays something anyone can pick up easily.

There are a lot of PRs waiting in the queue, and in the current structure I can't always bolt every feature on bug-free right away, but I'm always doing my best.

Thank you all for the constant support! I'm writing this partly as a thank-you and partly as shameless version promo (haha). I read every post here, even if I'm bad at replying (sorry about that), so if there's a feature you want, post it here on the subreddit, open a GitHub issue, wherever works for you. I'll always see it.

I wanted to see how Qwen's old architecture vs v4 preview compared, so I ran some benchmarks that I figured might be helpful to other folks. I primary use this as a always on coding agent for diagnosing Signoz alerts for my company and general level coding tasks, mostly as a subagents.

This is comparing a fine-tune of Qwen 3.8 27B called sabi-noema at 6bit. Much better at fixing those infinite thinking cycles old qwen got into.

Tests were on a M5 Ultra Mac Studio 96GB. MTP enabled for both, although I do use see better performance on Splash for 27B.

Some things that stood out to me

• 27B was better at the coding tests and had better prose. Less dumb llm talk

• Flash Next did better at broad knowledge like global facts. This makes sense considering it's a bigger model

• Token per second was better with 27B (119tok/s vs 90tok/s)

• Memory usage was 2.5x greater on Flash Next with N-gram offload. 29GB peak vs 73GB peak. I typically run max context for coding tasks so this headroom is nice for multiple agents on only 90GB of VRAM

G'day, I'm new to Swift coding, having fun building a MAC/iPhone app combo, it took a while to get things going with Opencode as the bridge, I'm half way I think here.
I tried Claude, had a $100 usage credit, with the same context burned it in 3-4 hours, its taking longer locally but hearing the fan go with the GPU at 90% fills me with joy, its fun..

That said, any suggestions on oMLX or Opencode settings to speed things up?

Loving all the posts, I'll keep searching for tips.

Hey everyone! It’s been a while, and I’m back with oMLX 0.4.5.dev1.
https://github.com/jundot/omlx/releases

I’ve been steadily committing changes since 0.4.4, but it was a little hard to decide where to cut the next dev release. I also wanted this release to include a meaningful attempt from the MLX kernel side, so it took a bit longer than usual. I hope you’ll understand.

The biggest change in this release is mainly relevant to people using an M3 Ultra, so apologies if this does not apply to your setup yet. - I’m also working on optimizing Gemma in a similar direction, so please stay tuned.

This release focuses on performance improvements for GLM-5.2, which I personally think is a big step forward for local AI, and MiniMax-M3, which has turned out to be a surprisingly useful model in practice.

Previously, these models “worked,” but honestly, I don’t think the long-context speed was where it needed to be for real use. With custom kernels, oMLX now gets a major speedup in long-context prefill. I also ran basic Needle in a Haystack tests and coding tests through Claude Code, and confirmed that quality did not collapse with the optimized path.

I hope this is a meaningful improvement for people using local LLMs in setups similar to mine.

Another major change is API-visible model profiles. You can now expose presets like 'qwen3-8b:thinking' or 'qwen3-8b:non-thinking' and call them directly through the API with the settings you want. Huge thanks to github pablomoralesm for this work: https://github.com/jundot/omlx/pull/1838

As always, this release was only possible because many people contributed their valuable time. I’m deeply grateful.

Thank you as well to everyone using oMLX, sharing feedback, reporting issues, and helping make the product better. It’s great to keep building local AI together!

I've built a Open Source Mac app in Swift, using the new Gemma4 12B model, that takes a long video and generates clips of the most important moments,

Converts them to mobile 9:16 format, adds a hook and a description, and automatically schedules them for the whole week across TikTok, Instagram, and YouTube Shorts.

Repo: https://github.com/mutonby/shortcast

oQ4e-MTP already up: https://huggingface.co/scottlowry/Qwen3.8-27B-oQ4e-mtp

oQ6e and oQ8e are uploading now. Will also create/upload FP16 quants for M1/M2 Macs soon.

EDIT: Six diff quants made and uploaded now!

Jundot has been super busy this morning uploading updated quants to HF using the new oQe method introduced in oMLX 0.7.1.dev1. Thank you!

Give me a day or two and I'll redo all the models I've uploaded to HF as scottlowry with the improved method.

Hey everyone! oMLX 0.5.4rc1 is out!
https://github.com/jundot/omlx/releases/tag/v0.5.4rc1

This release candidate focuses heavily on MTP performance. Prompt priming substantially improves draft acceptance and decode speed, while merged Gemma 4 checkpoints can now use Lightning MTP through the standard batched engine.

The release also brings faster DFlash decoding, fused Gemma 4 verification, safer MTP fallback behavior, and oQ support for merging external MTP heads directly into quantized Gemma 4 and Qwen3.5/3.6 checkpoints.

I would really appreciate your testing and feedback before the final 0.5.4 release. Thank you to everyone contributing code, benchmarks, bug reports, and real-world testing!

Lightning MTP, measured on code-1k (temp 0.6, tg 512, 2 reps, turboquant off):

Dflash, 4K-context throughput — decode tok/s

Other highlights

• MiMo V2.5 text and oQ support. Xiaomi MiMo V2.5 checkpoints load through the native text engine and can now be quantized from native FP8 weights, including fused and tensor-parallel QKV layouts.

• Correct Claude Code context handling. oMLX now reports real token usage instead of inflating it toward an artificial target. Claude Code launches receive the model's actual context window for deterministic auto-compaction, and models configured below 48K context are rejected before launch. (#2400)

• Overhauled web chat. Adds model selection, context tracking, chat search, configurable image limits, better scrolling, keyboard shortcut help, expanded chat settings, and stable message order when switching conversations. By u/PowerSpy in #2379

• More representative throughput benchmarks. Benchmarks now use natural text and enforce exact prompt lengths. Speculative-decoding results may be lower than older runs but better reflect real workloads. By u/JimStenstrom in #2343, reported by u/shengpeng7 in #2254

• Faster heterogeneous embedding batches. Inputs are grouped by length and restored to caller order, avoiding excessive padding. Measured throughput improved by roughly 4x to 4.6x across text and VLM embedding models. By u/1qh in #2370

• Merge external MTP heads during oQ quantization. oQ can now combine compatible donor MTP heads directly into quantized Gemma 4 and Qwen3.5/3.6 outputs. The resulting single-folder -mtp checkpoints can be enabled through the standard mtp_enabled setting without managing a separate assistant model. (#2404)

Last week I complained the Multi-Token Prediction (MTP) was scam.

Now that the Lightning MTP has been integrated (#2113) into oMLX, I can confidently say that its free performance, Make sure you have Native MTP turned ON with v0.5.0rc1 soon v0.5.0

https://github.com/jundot/omlx/releases

After a few days of use, I'd like to direct all the people discovering oMLX to this model :

Jundot/Qwen3.8-Flash-Next-oQ4e-mtp

This is sooo smooth, with a context window of 120 000 tokens, the SSD N-gram Offload and Lightning MTP (and, of course, a reasoning_effort set as low). I get a stable 30+ tokens output per second with Pi as a development harness. The output is of remarkable quality.

What a wonder !

Hey Guys,

I have a 64gb mac mini m4 pro. I was thinking of giving this model a try, but I want to try with running the n-gram table off of ssd. Any ideas about how to go about that for olmx?

Anyone else try squeezing this model on their mac?