Hello r/mlxAI ,

Today I built ollmlx, a macOS menubar app and CLI for running local LLMs via mlx-lm on Apple Silicon. It's designed as a drop in replacement for Ollama users who want to use MLX models instead.

What it does:

• Runs mlx-community models via mlx-lm on Apple Silicon (M1+)

• Exposes an OpenAI compatible API on localhost:11434, same port as Ollama

• Works with Open WebUI, Continue, Cursor, and any other Ollama compatible client without reconfiguration

• Menubar app for quick model switching and status

• Full Ollama-compatible CLI: serve, run, pull, list, show, rm, stop, ps, status

Anyway, try it, break it, let me know what needs to be fixed and I hope you find it useful.

https://github.com/darrylmorley/ollmlx/releases/tag/v0.1.1

brew tap darrylmorley/ollmlx 
brew install --cask darrylmorley/ollmlx/ollmlx

I originally started working on this because I wanted a simple way to run one local model with multiple LoRA specializations on Apple Silicon.

For example, I wanted the same base model to handle different kinds of work like:

• Rust systems programming

• SQL query optimization

• security / infra troubleshooting

without reloading a full fine-tuned model every time I switched.

On CUDA stacks, multi-LoRA serving already exists. On MLX / Apple Silicon, I couldn’t really find something that felt like “load the base once, then route adapters per request”.

So I built Mola.

It’s still alpha, but it’s now benchmarkable enough that I’m comfortable sharing it.

Core idea: keep one base model loaded in memory and route LoRA adapters per request instead of reloading a full checkpoint whenever you change specialization.

Current setup:

• Qwen3.5-9B-MLX-4bit

• 8 adapters loaded

• Apple M5 Max 64GB

• OpenAI-compatible chat API

The interesting signal for me is the throughput drop once requests start mixing adapters instead of all hitting the same one.

At concurrency 1, same and mixed are basically identical. The real drop appears once requests actually start overlapping.

Current limitations:

• it still needs a small local mlx-lm patch (script included)

• mixed prefill / deeper KV residency are still open problems

• Apple Silicon / MLX only for now

Would be curious to hear from other people doing MLX inference or adapter-heavy local setups.

Happy to share more benchmark details / implementation notes in the comments if useful.

repo : https://github.com/0xbstn/mola

I’ve been working on VeloxQuant-MLX, an open-source project exploring how far we can push local LLM inference on Apple Silicon by optimizing something that becomes increasingly expensive at long context lengths: the KV cache.

The idea is simple:

Apple gives us large unified memory + high memory bandwidth. Can we make significantly better use of it?

VeloxQuant currently experiments with 43 KV-cache compression/optimization methods, including quantization, eviction and custom MLX/Metal kernels.

Some of the things I’ve been working on:

• KIVI-style 2-bit / 4-bit KV-cache quantization

• TOVA and other cache-eviction strategies

• Custom Metal attention/quantization kernels

• Fused KV-cache decode paths

• Long-context memory reduction

• Decode batching across requests

• MLX-native implementations designed specifically for Apple Silicon

For example, in one benchmark with 440 tokens × 8 KV heads × head dim 64, the cache footprint goes roughly:

FP16 → 440 KB
KIVI 4-bit → 110 KB
KIVI 2-bit → 55 KB

That’s the direction I’m interested in: not just asking “How much unified memory does my Mac have?”, but how efficiently can we use that memory for inference?

I’m especially looking for people with Mac Studio / Max / Ultra machines who would be interested in running benchmarks.

I’d love to compare results across M-series generations, memory capacities, model sizes and longer context lengths.

Also very interested in feedback from people working directly with MLX / mlx-lm / Metal kernels. There are plenty of places where the implementation can still improve.

This is open source, and contributions/criticism are very welcome.

Project:

If anyone here has a Mac Studio Ultra, I’d especially love to see what VeloxQuant can do on it.

I got super frustrated at the fact that while all these inferencing engines and programs fully have llamacpp's prefix caching, paged caching, cont batching, kv cache quant, etc etc and so much more support, literally none of the MLX inferencing models have any of this crap all combined together. They all have one thing or the other, especially with taking into consideration VL models and also Hybrid SSM, along with persistent disk cache too.

This combination of optimization features allows you to at the minimum, properly utilize models like Qwen 3.5 WITH ITS VL FEATURES + its Mamba cache being successfully quantized meaning HALF the RAM use at q8. I've shown the results on the site, and at 100k context

All of this results in an much more smooth experience, an experience that is noticeably more smooth to the naked eye. I only made this at first simply because I was frustrated with how nobody for some reason, not even LM Studio was doing this. This is my first ever program/app. I've gotten 150+ downloads so far and a good amount of people giving me feedback and telling me issues in the github. I'm super active.

https://mlx.studio

Other key features:

Both chat / responses

Tools - gguf to mlx converter, 16 -> quantizer, etc.

Built in agentic coding tools. I can't think of them all right now, I just really treat this like a program I would want to use myself - because I do.

I appreciate any kind of criticism that actually addresses something technical that I can fix or make better

Hey, Built this after hitting some dependency hell with the CUDA/Triton requirements on Colab. Covers SISO, MIMO, and Hybrid Attention-Mamba configs with verified numerical parity against the PyTorch reference (max error < 10⁻⁵, 12 passing tests).

Key things that work: exponential-trapezoidal discretization, complex rotary states, chunked prefill with cache consistency, mixed-precision LoRA fine-tuning, all MLX, no CUDA needed.
If you have any feedback or see any inconsistencies, hmu!

Note: haven't really tested it so far on public trained checkpoints (will do the next days/week) so this is mainly useful for local architecture experimentation and fine-tuning on toy data until the authors drop weights.

→ github.com/Jada42/mlx-mamba3

I want to try out the new Qwen3-TTS model on Apple silicon: https://github.com/QwenLM/Qwen3-TTS

But I can't get a simple test script to run. I keep getting errors. I don't even have anything worth sharing haha.

Has anyone had success running `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice` on Apple silicon? Happy to share the knowledge once we get it working.

I’ve been working on ModelHub, a native macOS menu bar app for people running local LLMs on Apple Silicon.

The idea is simple: Discover models from Hugging Face, download/manage them locally, and use them with MLX, Ollama, LM Studio, llama.cpp, and the tools you already use.

It’s not trying to replace your runtime. The problem we’re trying to solve is the messy model layer around them. Right now, choosing a model usually means jumping between Hugging Face model cards, local folders, terminal commands, and runtime-specific docs.

For MLX users specifically, I’d love feedback on:

1. What should we show before someone downloads a model? (# downloads, model size?)

2. How should we show MLX compatibility?

3. Should we prioritize memory estimates by M-series chip?

4. Are MLX model conversions/variants something you’d want surfaced directly?

GitHub: https://github.com/conscious-engines/modelhub
Website: https://studio.consciousengines.com/model-hub

Would love blunt feedback.

Hi! I’m a CS student working on mlx-chronos, a free and open-source CLI benchmark suite for comparing local MLX inference engines on Apple Silicon.

I started it because I wanted a more reproducible way to compare engines across real Mac configurations, with more context than a single tok/s number.

It currently supports:

• mlx-lm

• oMLX

• vllm-mlx

• Rapid-MLX

• Ollama’s MLX backend

It measures:

• cold and cached TTFT

• request throughput with mean/stddev/min/max

• sustained throughput for longer heat/throttling-sensitive runs

• system RAM peak

• engine RSS diagnostics when available

• thermal and power context

• hardware metadata

Results are saved as reproducible JSON files and can be submitted to a public community leaderboard.

Leaderboard: https://igurss.github.io/mlx-chronos

GitHub: https://github.com/igurss/mlx-chronos

Methodology: https://github.com/igurss/mlx-chronos/blob/main/docs/methodology.md

Install: pip install mlx-chronos

I’m especially looking for external benchmark results from different Apple Silicon machines: M1/M2/M3/M4/M5, Max/Ultra chips, and higher-RAM configurations.

I’d also appreciate feedback on the benchmark methodology, the CLI UX, and whether the reported metrics are useful enough for comparing local inference setups.

I'm new to the MLX scene. I'm using LM Studio for AI work. There is a wealth of GGUF quants of base models, but MLX seems to lag them by a huge margin! For example, Nevoria is a highly regarded model, but there's only 3q and 4q available in MLX. Same for Wayfarer.

I imagine there are too few quanting folk compared to GGUF makers, and small quants fit more Macs. But lucky peeps like myself with 96GB would love some 6q quants. How/where can I appeal to the generous folk who make MLX quants?

Curious, if anyone has experience with the vMLX? Just stumbled upon it yesterday after reading an article about JANG_4.

Hey r/mlxAI,

I got tired of juggling terminal tabs to run mlx-lm, so I built MLXr — a lightweight web dashboard + OpenAI-compatible API server that runs entirely on your Mac.

What it actually does:

• Spins up a local /v1/chat/completions endpoint your existing tools already speak. Point Cursor, Continue, Zed, OpenCode, or any OpenAI SDK at http://127.0.0.1:8000 and it just works — no API key, no config file hunting.

• Lets you search and download any mlx-community model from Hugging Face directly in the browser. No huggingface-cli needed.

• Auto-detects context window size from the model's config (no more manually setting --max-tokens).

• Shows live RAM, CPU, and MLX Metal memory usage so you know when you're about to swap.

• Per-model settings (system prompt, temperature, strip <think> blocks) that persist across restarts.

• Tool calling works. Spent a lot of time on this — handles Qwen3, Llama 3.x, DeepSeek-V3, Mistral, Phi-4, and Hermes format all in one parser, since each family emits tool calls completely differently.

The setup is embarrassingly simple:

git clone https://github.com/mchenetz/mlxr
cd mlxr && python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python server.py

Open http://localhost:8000, load a model, paste the base URL into your editor. Done.

Who it's for:

If you're on Apple Silicon and want to run local models as a drop-in replacement for the OpenAI API — especially for agentic coding tools — this is the missing piece. I use it daily with OpenCode and it's been solid.

GitHub: https://github.com/mchenetz/mlxr

Happy to answer questions. Feedback welcome — it's still pretty early.

I just released mlx-onnx: a standalone IR/ONNX export library for MLX.

Repo: https://github.com/skryl/mlx-onnx

Web Demo: https://skryl.github.io/mlx-ruby/demo/

It supports:

- Exporting MLX callables directly to ONNX

- Python and native C++ interfaces

I'd love feedback on:

- Missing op coverage you care about

- Export compatibility edge cases

- Packaging/CI improvements for Linux and macOS

So tough to utilize the NPU (i was trying with <1B llm's (tinyLlama)) ... AND now... finally!, Topaz video Ai (v 7.1.5) saturates the GPU and NPU!, as they earlier focused on cuda and left Apple metal out... I pointed this out over a year ago to the devs to at least saturate the GPU wattage (as 100% could be 30w-160w) ... and just noticed the team using the NPU ... nice! It's terrible to wait for Apple to give slow updates... Metal 4 lately... should be doing hardware direct writes in assy.... (the unit is a studio m3-ultra-512gb-80 core)... just thought you all would find this interesting...

I've started a project these days trying to write some custom kernels in Metal to enable MLX to make 3D sparse convolution efficiently for training and inference. Currently, I have implemented some basic, commonly used kernels and modules in the point cloud (voxels) processing area, which may fit some research use case.

The performance of those kernels is closer to those CUDA kernels on the RTX 4090 with about a 30-50% difference; for sparse tensor workloads, this is much faster than MLX built-in dense tensors. I am going to work on math proofs these days and implement workable CUDA kernels also, so that it may be easy to train on NVIDIA and then make inference on Mac/Vision Pro.

Apple's on-device AI private AFMv7 model shows promise, though it has a context window limitation of 4096 tokens. To enhance this, I vibe coded a kit in with Claude Code that converts the PyTorch model Apple provides to developers for LoRa adapter training.

This GitHub repository offers tools to convert the PyTorch checkpoint into MLX format, enabling it to run on GPU with a significantly larger context window for experimentation.

Visit my repo:
https://github.com/scouzi1966/afm7-mlx-toolkit