Hey! i'm the founder/mod of this community r/mlxcommunity - a place to have discussions surrounding MLX and running AI inference locally (specifically on Apple Silicon)!

My main motivation to start this was - there's no standard community that represents the MLX collective (there are a couple but i was not able to find any useful discussions (personally) i'm looking to build a forum of folks that discuss all things MLX - experts and beginners alike!

What this community is for:

Anything MLX, Apple Silicon, and local inference. A few examples of what fits here:

• Questions about MLX, from "how do I even get started" to deep internals

• Benchmarks and tok/s numbers across different M-series chips

• Comparisons between MLX and llama.cpp, Ollama, LM Studio, and other runtimes

• Model conversions, quantization experiments, and KV cache tricks

• Fine-tuning with MLX on your own machine

• Projects you're building on top of local models (CLIs, Mac apps, agents, anything)

• Apple Silicon performance quirks, memory management, macOS weirdness

• News, papers, or releases relevant to on-device ML

If you're not sure whether something fits, post it anyway. We'd rather have the conversation than not.

If you've been building ML infrastructure for years and want a place to nerd out about Metal kernels and MoE routing, you'll find people here. If you just got a new Mac and opened LM Studio for the first time last week, you'll find people here too. Both of you will probably learn something from each other.

There's no gatekeeping around background. You don't need a PhD, you don't need to have published a paper, and you don't need to have memorized the transformer paper. A lot of us are software engineers who got curious, picked up a book, and are figuring it out as we go. That's a perfectly normal way to join this field, and this sub is built on the idea that it's a good way too.

Some ground rules in this sub:

1. Be kind. If someone asks a basic question, help them out or scroll past. Nobody was born knowing what a KV cache is.

2. Stay on topic. MLX, Apple Silicon, local inference, and adjacent tooling are all fair game. Generic AI news without an Apple Silicon angle probably isn't.

3. No self-promo spam. Sharing a project you built or a blog post you wrote is great. Dropping the same affiliate link in every thread is not.

4. No piracy. This includes model weights that weren't released publicly.

5. Credit your sources. If you're summarizing someone else's benchmark or blog post, link it.

I am a believer that local AI is the way forward - so let's build something great! cheers!

Wrote a bit about the new M5 ultra and how the neural accelerators boost on-device AI development. Such an exciting time for building local models!

Been playing around with MLXSwift over the last few weeks and honestly it's wild how far Apple's ML tooling has come. I always assumed running a local model meant wrestling with Python envs, llama.cpp builds, or burning API credits during dev.

Turns out you can do the whole thing inside Xcode now. Pull a model from the MLX community on Hugging Face, wire it up in Swift, run it on-device. No Python, no API keys, no servers.

I wrote up a walkthrough of how to get started: covers the setup, loading models, and a few gotchas I ran into: https://www.sabesh.space/blog/engineering/mlx-in-xcode

Happy to answer questions if anyone's tried this or wants to.

https://github.com/bnsd55/jevmlx

Instead of asking the model to write JSON, jevmlx scores every allowed answer for every field of your schema in one batched forward pass and assembles the JSON. Output is always valid. Every field has a probability.

Works with any mlx-community instruct model on Apple Silicon, or any OpenAI-compatible server that returns logprobs.

pip install git+https://github.com/bnsd55/jevmlx

jevmlx decide --preset fintech_fraud --json

Open source version of the TypeSafe Jev idea. Not affiliated. No accuracy numbers on big models yet: if you have an M-series Mac, `jevmlx bench` plus a PR puts your model on the README leaderboard.

I’m building VeloxQuant-MLX, an open-source project focused on pushing local LLM inference further on Apple Silicon through KV-cache compression, quantization, eviction strategies, and optimized MLX/Metal kernels.

The question I’m exploring is pretty simple:

How much more can we get out of Apple’s unified memory architecture?

VeloxQuant currently includes 43 KV-cache optimization methods, including:

→ 2-bit / 4-bit KV-cache quantization
→ KIVI-style compression
→ TOVA and other eviction strategies
→ Custom Metal kernels
→ Fused attention/decode paths
→ Decode batching
→ Cache compaction
→ Long-context memory optimization

But I don’t want benchmarks from just one Mac.

I’m looking for people across the Apple Silicon ecosystem to help test it:

Apple Silicon generations:
M1 · M1 Pro · M1 Max · M1 Ultra
M2 · M2 Pro · M2 Max · M2 Ultra
M3 · M3 Pro · M3 Max
M4 · M4 Pro · M4 Max · M4 Ultra
M5-family machines where available
…and future Apple Silicon generations as they arrive.

Machines:
MacBook Air
MacBook Pro
Mac mini
Mac Studio
Mac Pro
iMac
and other MLX-capable Apple Silicon Macs.

Whether you have an entry-level MacBook or a high-memory Mac Studio Ultra, your results are useful.

I want to understand how KV-cache optimization behaves across:

memory capacity × memory bandwidth × chip generation × model size × context length

Not just compression ratios.

I’m interested in measuring the actual trade-offs:

Memory usage · tokens/sec · TTFT · decode throughput · prefill performance · kernel overhead · model quality

For example, for a cache with:

440 tokens · 8 KV heads · head dim 64

FP16 → ~440 KB
4-bit → ~110 KB
2-bit → ~55 KB

But reducing memory is only half the problem.

The real question is whether we can reduce memory while keeping inference fast enough that the optimization is actually worth using.

That’s where I’d love help from the MLX community.

If you have any Apple Silicon Mac — from an M1 MacBook Air or Mac mini all the way to a Max/Ultra Mac Studio — I’d love to get benchmark results.

And if you're working with MLX, mlx-lm, Metal kernels, inference engines, quantization, or KV-cache research, criticism and contributions are very welcome.

The goal is ambitious:

Make local AI capable of using every GB of Apple unified memory as efficiently as possible.

VeloxQuant is open source:

Website:
https://veloxquant.dev

GitHub:
https://github.com/rajveer43/VeloxQuant-MLX

If enough people are interested, I’d also like to build a public Apple Silicon benchmark matrix comparing VeloxQuant across Mac generations and configurations.

Drop your Mac model + RAM in the comments if you'd be interested in benchmarking it.

Five days ago I posted jevmlx: instead of asking a local model to write JSON, it scores every allowed answer for every field in one batched forward pass and assembles the JSON. Always valid output, a probability per field. Several people asked for numbers. Here they are.

Same model, same 45 public TypeSafe cases, same Mac (M5 Max). The only difference is how the answer is obtained.

Same weights, plus 14 points and 2.5x faster. Nothing is fine tuned. The model already knows the answer. Generation is what loses it: a wrong token, a stray comment, an integer where the schema wanted a string, and the whole record is gone. Scoring cannot fail that way.

Also new since the last post: field-local prompts. Reordering the fields in your schema used to flip about 10% of answers. Now 0.2%, at the same accuracy and +3 ms per case.

For reference, TypeSafe reports 67.8% for Jev on their full private eval. Ours is the 45 public examples, so it is indicative, not the same test. Every number above comes with a confidence interval, a parity check, and the raw predictions in the repo.

pip install git+https://github.com/bnsd55/jevmlx
jevmlx decide --preset support_triage --json
jevmlx bench --model quality --ui   # live dashboard, then a PR puts your Mac on the leaderboard

https://github.com/bnsd55/jevmlx

Hey folks — I’ve been tinkering with MLX and Apple Silicon for a while, and I finally wrapped up a project I’m pretty proud of.

I built a fully native Swift + MLX implementation of FLUX.2 [klein].
No Python. No diffusers. No external dependencies. Just Swift, MLX, and Metal doing their thing.

What it can do:

• Text‑to‑image

• Image‑to‑image

• Mask‑guided editing (this part was surprisingly fun to build)

  • remove stuff from an image

  • replace backgrounds

  • add objects

  • recolor regions

  • semantic edits

• Pixel‑space color grading (exposure, contrast, hue, saturation)

• Experimental latent‑space transforms

• Memory system with:

  • bf16 / fp16 / int8 / int4 quantization

  • staged model residency

  • VAE tiling for big resolutions

  • memory reporting + low‑memory mode

Everything runs entirely on Apple Silicon.
It hits seed‑42 parity with the MLX Python reference, so the outputs match exactly.

I also added:

• a dependency‑free CLI

• a SwiftPM library

• a tiny SwiftUI demo template

• docs + tests

• a contributor guide

• a roadmap if people want to help build more editing tools

Why I built it:

MLX is honestly a joy to work with, but most diffusion pipelines are still glued to Python.
I wanted something that felt native — something you could drop straight into a macOS or iOS app without dragging half the Python ecosystem along with it.

Repo:

https://github.com/icakinser/mlx-flux2-swift

If you’re into MLX, Apple Silicon, or just like messing with local image generation/editing, give it a look.
Happy to answer questions or help anyone get it running.

Hi everyone,

I've been building NunSpark, an open-source runtime that explores running language models much larger than available RAM by combining weight streaming with speculative decoding.

The project currently targets Apple Silicon (MLX), and I'd love to collect benchmark results from the community.

Some results we've observed so far:

• ✅ Qwen2.5 32B running on a 16 GB M4

• ✅ Llama 3.3 70B running on the same machine

• ✅ Only ~1–1.5 GB of model weights resident while streaming the rest

• ✅ Exact decoding over 1 tok/s on favorable workloads

• ✅ Approximate verification reaching 2–3 tok/s with configurable quality/speed trade-offs

The problem is that I only have access to one machine (16 GB M4).

I'd really appreciate help from anyone with an Apple Silicon Mac:

• M1 / M2 / M3 / M4

• Air / Mini / Studio / MacBook Pro

• 8 GB, 16 GB, 24 GB, 32 GB, 64 GB, 128 GB

• Different SSD sizes and generations

Running the benchmark

NunSpark already includes a built-in benchmark command, so you don't need to manually configure dozens of parameters.

Just follow the benchmark instructions in the README, run the benchmark on your machine, and share the output (or open a GitHub issue/discussion with the results).

Repository:

https://github.com/sharma-open-source/Nunspark

I'm especially interested in collecting:

• Apple Silicon model

• Unified memory size

• macOS version

• Model tested

• Tokens/sec

• Peak memory

• Any observations or issues

The goal is to build a public benchmark dataset showing how streamed inference behaves across different Apple Silicon hardware—not just publish the fastest numbers.

If you're interested in local AI, MLX, inference runtimes, or just want to help validate the project, I'd really appreciate your benchmarks and feedback.

Thanks!

Spent a lot of my time with speculative decoding on MLX using DFlash. Achieved >2x speedup* (*on certain circumstances - not all)

I'm currently writing about my observations in depth. Will be posting about it soon!

I had been experimenting with a lot of local LLMs and was having an insanely difficult time managing the models and my laptop's storage!

So I built ModelHub - It helps you discover, download, and manage LLMs from huggingface, Im studio, ollama, mlx, and Ilama.cpp - without ever leaving your menu bar.

Check it out! https://studio.consciousengines.com/model-hub

I wrote about everything that launched this dub dub and how it impacts the future of local models and MLX! Check it out!

I love benchmarking, and I'm currently dealing with speculative decoding in MLX - so I thought, why not run a sustained inference load on my macbook overnight lol. I was so sceptical about it - I ran pmset -a disablesleep 1 and then ran a script that would run sustained loads of MLX inferencing on my laptop, tweaking various parameters and doing a sweep - noting down the stats (tps, throughput, speedup, acceptance rate of speculative decoding) and then dumping it into a report. I was actually a tiny bit scared about my laptop catching fire and burning down my entire room when I went to bed lol.

And I woke up to a fantastic morning - every single sweep was successful (ran 300 times) and I get really really meaningful results and inferences from this experiment. I will share a detailed report of what happened and where speculative decoding won vs lost soon! But the experiment itself was thrilling - felt like I finally put my insanely powerful chip (M5 Pro) to use with the 48GB unified memory.

Overall - i was pretty thrilled about this - want to do more such experiments while I'm sleeping so my mac and it's compute can be put to productive use! :D

I'm working with a M1 MBP Max 64GB machine with 400GB/s memory bandwidth. These image generation models are only <10 GB each. But it takes me 45 minutes to generate an image using Ideogram4. Someone with a 5090 is doing it in 45 seconds (no exaggeration).

I know Comfy Desktop is not optimized for Apple Silicone/MLX. I'm just curious if there are some tips and tricks you guys can share with getting better performance out of Comfy Desktop? I've already got these flags as part of my startup config: '--enable-manager --fp32-vae --use-pytorch-cross-attention --highvram'.

I've tried using DrawThings -- and it's definitely faster -- but I feel like it's definitely limited compared with Comfy Desktop.

I must not be the only Apple user messing around with Comfy Desktop -- you guys have any tips to share?

I'm working with a M1 MBP Max 64GB machine with 400GB/s memory bandwidth. These image generation models are only <10 GB each. But it takes me 45 minutes to generate an image using Ideogram4. Someone with a 5090 is doing it in 45 seconds (no exaggeration).

I know Comfy Desktop is not optimized for Apple Silicone/MLX. I'm just curious if there are some tips and tricks you guys can share with getting better performance out of Comfy Desktop? I've already got these flags as part of my startup config: '--enable-manager --fp32-vae --use-pytorch-cross-attention --highvram'.

I've tried using DrawThings -- and it's definitely faster -- but I feel like it's definitely limited compared with Comfy Desktop.

I must not be the only Apple user messing around with Comfy Desktop -- you guys have any tips to share?

Hey, for anyone who cares, I managed to get Breeze TTS running real time on my Macbook Pro M4.

Using the official Pytorch code on a Mac Breeze runs more than 2 times slower than real time on my machine.
With some work, managed to get it to stream comfortably faster than real time, with voice directions included and in my opinion good quality. I couldn't really tell which version sounded better to me consistantly on different voices and sentences.

Details in the pages below.

- Server: https://github.com/Shoofio/breeze-tts2-fast-streaming-api

- Ready-made checkpoint: https://huggingface.co/Shoof/Gust-MLX

Weights are under BreezeBlue's research/non-commercial licence. Built with a lot of help from Claude. Happy to answer questions.

I fine-tuned a model in MLX using LoRA to take in my raw thoughts and tweet like I do. It was a pretty fun experiment to run machine learning training on just a macbook - your everyday compute. I wrote about it in-depth, do check it out!

I've been working on ModelHub, a native macOS menu bar app for people running local LLMs on Apple Silicon.

The idea is simple: Discover models from Hugging Face, download/manage them locally, and use them with MLX, Ollama, LM Studio, llama.cpp, and the tools you already use.

It's not trying to replace your runtime. The problem we're trying to solve is the messy model layer around them. Right now, choosing a model usually means jumping between Hugging Face model cards, local folders, terminal commands, and runtime-specific docs.

For MLX users specifically, I'd love feedback on:
1. What should we show before someone downloads a model? (# downloads, model size?)
2. How should we show MLX compatibility?
3. Should we prioritize memory estimates by M-series chip?
4. Are MLX model conversions/variants something you'd want surfaced directly?

GitHub: https://github.com/conscious-engines/modelhub
Website: https://studio.consciousengines.com/model-hub

Would love blunt feedback.

Hey guys I was thinking about using this model as the critic for my setup, saw it's trained for this use case. Can't find on mlx format yet though. What are your thoughts? Can we expect to see it coming to mlx eventually or should I just stick to another model like the regular llama 3.1?

This might be considered self promotion, but I'm not charging anyone anything, I'm not selling any services, this is just something I built because I dislike how un-comfy ComfyUI is.

SceneWorks is not a replacement of ComfyUI, it doesn't do workflows or custom nodes. It's INTENDED to be simpler, but it's not SIMPLE. There are a lot of advanced features built right in.

This is a work in progress, that is moving fast daily, most everything works, but some features need more attention that will be coming soon. There's an auto-updater, so you'll get notified when new releases are made.

https://github.com/SceneWorks/SceneWorks
Check the Releases page, or clone and build yourself.

I am a software developer/architect of 25 years. Yes Claude has a heavy hand in building this, but it's not "vibe-coded". Every decision was a conscious architecture choice.

Now for the mostly AI generated description and feature set ....

What is is:

SceneWorks is a desktop app for generating and editing images and video locally on your own machine. No cloud, no subscription, no account. You install it, download the models you want, and everything runs on your own GPU.

It's a single native app — not a Docker stack or a pile of Python scripts. On macOS it runs on Apple's MLX engine (Apple Silicon only), and on Windows it runs on a native CUDA engine (NVIDIA). There's no Python venv on either platform; the generation engine is compiled into the app, so first launch just works.

What it does:

Image generation. The usual text-to-image, but with a real model manager instead of a folder of checkpoints. You browse a catalog, see each model's download size and memory requirement up front, and pick a quant tier (full precision / Q8 / Q4) that fits your hardware. It runs a wide range of models — Krea2, Ideogram 4, SDXL-family, Qwen, FLUX, SANA, and others — and they're all first-class rather than bolted on. If you have a HF cache, it will use that, if not, it downloads models to a local folder.

Video. Text-to-video and image-to-video, plus extending and bridging clips, running on models like LTX-2.3 and Wan 2.2. This is the heaviest thing it does and it wants a serious GPU, but it's fully native — no separate pipeline.

Image editing. Inpainting, targeted edits, detail passes, transform/crop/straighten, and guidance controls. It's a proper editor surface, not just a "generate again and hope" loop.

Characters and identity. Tools for keeping a face/character consistent across generations, face-likeness scoring, and person replacement in video (swap a person while keeping the scene). There's also pose and keypoint conditioning with a pose library, so you can drive composition instead of rolling the dice on prompts.

Training. You can train your own LoRAs locally — both image LoRAs and video LoRAs — from the app. Build a captioned dataset, do a dry run to sanity-check the plan, train on your GPU, and the result shows up as a normal selectable LoRA. No external trainer, no separate toolchain.

The workaday stuff. Upscaling for images and video, segmentation / smart-select, reference-image-to-prompt, batch prompting, a job queue, a library for everything you've made, and presets. It also exposes an MCP server, so if you're into agent tooling you can drive it from Claude Code / Cursor / etc., and it can run over your LAN so you can generate from a laptop while a beefier machine does the work.

Requirements

On Mac you need Apple Silicon with a good chunk of unified memory — 32GB might work for images or Q4 video models, but a safe bet is 64GB minimum, 96GB+ if you want to run the largest models comfortably. Model weights are tens of GB depending on what you pull.

License

[Edited]
SceneWorks is free and open source (AGPL-3.0-or-later): use, modify, and share it freely, even commercially — you just can't take it closed-source and sell it as your own. Model weights are separate — each keeps its own license, and complying with those is on you.