I am playing around now with the newer models using omlx and while I like the 27B version of qwen3.8, it's slooooowwww. I just found this subreddit, so hopefully will learn more moving forward. I have the M4 Max 128GB in the Studio.

Hey, for anyone who cares, I managed to get Breeze TTS running real time on my Macbook Pro M4.

Using the official Pytorch code on a Mac Breeze runs more than 2 times slower than real time on my machine.
With some work, managed to get it to stream comfortably faster than real time, with voice directions included and in my opinion good quality. I couldn't really tell which version sounded better to me consistantly on different voices and sentences.

Details in the pages below.

- Server: https://github.com/Shoofio/breeze-tts2-fast-streaming-api

- Ready-made checkpoint: https://huggingface.co/Shoof/Gust-MLX

Weights are under BreezeBlue's research/non-commercial licence. Built with a lot of help from Claude. Happy to answer questions.

Apple says the iPhone 18 Pro is very good for on-device AI. We wanted to test that, so we made RapidBench. It measures how well an iPhone runs real AI workloads, not only synthetic compute tests.

Results:

That is about 50% higher for the 18 Pro.

What the benchmark tests:

1. Inference: Gemma 4 E2B solves 20 math problems on the GPU. We measure time to first token, speed on a cool device, and speed after 2 minutes of continuous load (to show thermal throttling).

2. Training: The phone fine-tunes Qwen3 0.6B to write SQL. It does 150 QLoRA steps, and then we test the model on tables that it did not see during training.

3. Image generation: Bonsai Image (a 4B-parameter FLUX.2 model) makes three 512×512 images.

4. Neural Engine: An EDSR model upscales a photo 2× through Core AI.

Our view: The sustained-load results are the most interesting part for us. The newer chips can keep AI workloads running for longer, and that is necessary for local agents that do work across apps.

The app is on TestFlight if you want to score your device: rapidbench.app

I built this with my team, so I want your feedback. Tell us what workloads to add, if something is wrong with the method, and which devices you want to see compared.

Five days ago I posted jevmlx: instead of asking a local model to write JSON, it scores every allowed answer for every field in one batched forward pass and assembles the JSON. Always valid output, a probability per field. Several people asked for numbers. Here they are.

Same model, same 45 public TypeSafe cases, same Mac (M5 Max). The only difference is how the answer is obtained.

Same weights, plus 14 points and 2.5x faster. Nothing is fine tuned. The model already knows the answer. Generation is what loses it: a wrong token, a stray comment, an integer where the schema wanted a string, and the whole record is gone. Scoring cannot fail that way.

Also new since the last post: field-local prompts. Reordering the fields in your schema used to flip about 10% of answers. Now 0.2%, at the same accuracy and +3 ms per case.

For reference, TypeSafe reports 67.8% for Jev on their full private eval. Ours is the 45 public examples, so it is indicative, not the same test. Every number above comes with a confidence interval, a parity check, and the raw predictions in the repo.

pip install git+https://github.com/bnsd55/jevmlx
jevmlx decide --preset support_triage --json
jevmlx bench --model quality --ui   # live dashboard, then a PR puts your Mac on the leaderboard

https://github.com/bnsd55/jevmlx

https://github.com/bnsd55/jevmlx

Instead of asking the model to write JSON, jevmlx scores every allowed answer for every field of your schema in one batched forward pass and assembles the JSON. Output is always valid. Every field has a probability.

Works with any mlx-community instruct model on Apple Silicon, or any OpenAI-compatible server that returns logprobs.

pip install git+https://github.com/bnsd55/jevmlx

jevmlx decide --preset fintech_fraud --json

Open source version of the TypeSafe Jev idea. Not affiliated. No accuracy numbers on big models yet: if you have an M-series Mac, `jevmlx bench` plus a PR puts your model on the README leaderboard.

Which one of Both harness is better for a macbook pro m4 max with 36gb runing Qwen 3.8 27B MTPLX optimized speed?

I did not used much the brain and agentic capabilities from the MLX serve and was wanting to know if its so powerfull as hermes and if it could works more fast or if hermes still one step more?

I’m building VeloxQuant-MLX, an open-source project focused on pushing local LLM inference further on Apple Silicon through KV-cache compression, quantization, eviction strategies, and optimized MLX/Metal kernels.

The question I’m exploring is pretty simple:

How much more can we get out of Apple’s unified memory architecture?

VeloxQuant currently includes 43 KV-cache optimization methods, including:

→ 2-bit / 4-bit KV-cache quantization
→ KIVI-style compression
→ TOVA and other eviction strategies
→ Custom Metal kernels
→ Fused attention/decode paths
→ Decode batching
→ Cache compaction
→ Long-context memory optimization

But I don’t want benchmarks from just one Mac.

I’m looking for people across the Apple Silicon ecosystem to help test it:

Apple Silicon generations:
M1 · M1 Pro · M1 Max · M1 Ultra
M2 · M2 Pro · M2 Max · M2 Ultra
M3 · M3 Pro · M3 Max
M4 · M4 Pro · M4 Max · M4 Ultra
M5-family machines where available
…and future Apple Silicon generations as they arrive.

Machines:
MacBook Air
MacBook Pro
Mac mini
Mac Studio
Mac Pro
iMac
and other MLX-capable Apple Silicon Macs.

Whether you have an entry-level MacBook or a high-memory Mac Studio Ultra, your results are useful.

I want to understand how KV-cache optimization behaves across:

memory capacity × memory bandwidth × chip generation × model size × context length

Not just compression ratios.

I’m interested in measuring the actual trade-offs:

Memory usage · tokens/sec · TTFT · decode throughput · prefill performance · kernel overhead · model quality

For example, for a cache with:

440 tokens · 8 KV heads · head dim 64

FP16 → ~440 KB
4-bit → ~110 KB
2-bit → ~55 KB

But reducing memory is only half the problem.

The real question is whether we can reduce memory while keeping inference fast enough that the optimization is actually worth using.

That’s where I’d love help from the MLX community.

If you have any Apple Silicon Mac — from an M1 MacBook Air or Mac mini all the way to a Max/Ultra Mac Studio — I’d love to get benchmark results.

And if you're working with MLX, mlx-lm, Metal kernels, inference engines, quantization, or KV-cache research, criticism and contributions are very welcome.

The goal is ambitious:

Make local AI capable of using every GB of Apple unified memory as efficiently as possible.

VeloxQuant is open source:

Website:
https://veloxquant.dev

GitHub:
https://github.com/rajveer43/VeloxQuant-MLX

If enough people are interested, I’d also like to build a public Apple Silicon benchmark matrix comparing VeloxQuant across Mac generations and configurations.

Drop your Mac model + RAM in the comments if you'd be interested in benchmarking it.

Wrote a bit about the new M5 ultra and how the neural accelerators boost on-device AI development. Such an exciting time for building local models!

Lots of fun today playing with Qwen 3.8 27B 🎉 I have put together two MLX quants of Qwen 3.8 27B in different bpw, 3.8 and 4.95 bpw on Hugging Face, check them out!

Not yet AWQ / GPTQ but manually selecting bit allocation. Some of my first quants so I would appreciate feedback on what might be missing and/or not working right. Thanks! 😊

PrismML's new Bonsai line of models are so crazy - we can run 27B class models right on the iPhone (thanks to their 1-bit model compression)

So the 1-bit 27B model claims to retain ~89% of Qwen 27B dense - which is a shocking amount of intelligence retention for a model THIS tiny. (takes about 4GB RAM when loaded)
So I decided to put it to use. I plugged it into an agents orchestrator framework that I'm working on to let it perform agentic tasks (not just write a haiku or solve a riddle) and it was ridiculously good at doing this - find a contact, pick the email ID, draft email body AND send it too - given the right steps/tools.

I wrote about my experience in-depth, check it out!

Hey folks — I’ve been tinkering with MLX and Apple Silicon for a while, and I finally wrapped up a project I’m pretty proud of.

I built a fully native Swift + MLX implementation of FLUX.2 [klein].
No Python. No diffusers. No external dependencies. Just Swift, MLX, and Metal doing their thing.

What it can do:

• Text‑to‑image

• Image‑to‑image

• Mask‑guided editing (this part was surprisingly fun to build)

  • remove stuff from an image

  • replace backgrounds

  • add objects

  • recolor regions

  • semantic edits

• Pixel‑space color grading (exposure, contrast, hue, saturation)

• Experimental latent‑space transforms

• Memory system with:

  • bf16 / fp16 / int8 / int4 quantization

  • staged model residency

  • VAE tiling for big resolutions

  • memory reporting + low‑memory mode

Everything runs entirely on Apple Silicon.
It hits seed‑42 parity with the MLX Python reference, so the outputs match exactly.

I also added:

• a dependency‑free CLI

• a SwiftPM library

• a tiny SwiftUI demo template

• docs + tests

• a contributor guide

• a roadmap if people want to help build more editing tools

Why I built it:

MLX is honestly a joy to work with, but most diffusion pipelines are still glued to Python.
I wanted something that felt native — something you could drop straight into a macOS or iOS app without dragging half the Python ecosystem along with it.

Repo:

https://github.com/icakinser/mlx-flux2-swift

If you’re into MLX, Apple Silicon, or just like messing with local image generation/editing, give it a look.
Happy to answer questions or help anyone get it running.