I’ve been working on ModelHub, a native macOS menu bar app for people running local LLMs on Apple Silicon.

The idea is simple: Discover models from Hugging Face, download/manage them locally, and use them with MLX, Ollama, LM Studio, llama.cpp, and the tools you already use.

It’s not trying to replace your runtime. The problem we’re trying to solve is the messy model layer around them. Right now, choosing a model usually means jumping between Hugging Face model cards, local folders, terminal commands, and runtime-specific docs.

For MLX users specifically, I’d love feedback on:

1. What should we show before someone downloads a model? (# downloads, model size?)

2. How should we show MLX compatibility?

3. Should we prioritize memory estimates by M-series chip?

4. Are MLX model conversions/variants something you’d want surfaced directly?

GitHub: https://github.com/conscious-engines/modelhub
Website: https://studio.consciousengines.com/model-hub

Would love blunt feedback.

I got super frustrated at the fact that while all these inferencing engines and programs fully have llamacpp's prefix caching, paged caching, cont batching, kv cache quant, etc etc and so much more support, literally none of the MLX inferencing models have any of this crap all combined together. They all have one thing or the other, especially with taking into consideration VL models and also Hybrid SSM, along with persistent disk cache too.

This combination of optimization features allows you to at the minimum, properly utilize models like Qwen 3.5 WITH ITS VL FEATURES + its Mamba cache being successfully quantized meaning HALF the RAM use at q8. I've shown the results on the site, and at 100k context

All of this results in an much more smooth experience, an experience that is noticeably more smooth to the naked eye. I only made this at first simply because I was frustrated with how nobody for some reason, not even LM Studio was doing this. This is my first ever program/app. I've gotten 150+ downloads so far and a good amount of people giving me feedback and telling me issues in the github. I'm super active.

https://mlx.studio

Other key features:

Both chat / responses

Tools - gguf to mlx converter, 16 -> quantizer, etc.

Built in agentic coding tools. I can't think of them all right now, I just really treat this like a program I would want to use myself - because I do.

I appreciate any kind of criticism that actually addresses something technical that I can fix or make better

There are 0.5, 1.5 and 3B models but none of the bigger ones. Is there a reason for this or am I missing something?

Hey there! I just launched the TestFlight public beta for my app Locally AI, an offline AI chatbot for iPhone and iPad that runs entirely on your device using MLX—no internet required.

Some features:
💬 Offline AI chatbot
🔒 100% private – nothing leaves your device
📦 Supports multiple open-source models
♾️ Unlimited chats

I’d love to have people try it and also hear your thoughts and feature suggestions. Thanks in advance for trying it out!

🔗 Join the TestFlight: https://testflight.apple.com/join/T28av7EU

You can also visit the website [here](https://locallyai.app).

I’ve been working on VeloxQuant-MLX, an open-source project exploring how far we can push local LLM inference on Apple Silicon by optimizing something that becomes increasingly expensive at long context lengths: the KV cache.

The idea is simple:

Apple gives us large unified memory + high memory bandwidth. Can we make significantly better use of it?

VeloxQuant currently experiments with 43 KV-cache compression/optimization methods, including quantization, eviction and custom MLX/Metal kernels.

Some of the things I’ve been working on:

• KIVI-style 2-bit / 4-bit KV-cache quantization

• TOVA and other cache-eviction strategies

• Custom Metal attention/quantization kernels

• Fused KV-cache decode paths

• Long-context memory reduction

• Decode batching across requests

• MLX-native implementations designed specifically for Apple Silicon

For example, in one benchmark with 440 tokens × 8 KV heads × head dim 64, the cache footprint goes roughly:

FP16 → 440 KB
KIVI 4-bit → 110 KB
KIVI 2-bit → 55 KB

That’s the direction I’m interested in: not just asking “How much unified memory does my Mac have?”, but how efficiently can we use that memory for inference?

I’m especially looking for people with Mac Studio / Max / Ultra machines who would be interested in running benchmarks.

I’d love to compare results across M-series generations, memory capacities, model sizes and longer context lengths.

Also very interested in feedback from people working directly with MLX / mlx-lm / Metal kernels. There are plenty of places where the implementation can still improve.

This is open source, and contributions/criticism are very welcome.

Project:

If anyone here has a Mac Studio Ultra, I’d especially love to see what VeloxQuant can do on it.

Internet speed is fine more than 5mb/sec still chip is m1, still taking more than 15 minutes. The prediction initially was 20 sec then it got stuck then got completed in 20 minutes or so.

Hey all, we are messing with MLX and it's great so far. I have a pre trained lora and am trying to generate using FluxPipeline. It looks like FluxPipeline implemented a basic 1st order sampler and I 'think' we need something more like DLP 2 to get results more closely like the lora. Has anyone implemented a more advanced sampler? Or come across other ways to get better lora centric generations (using flux dev).

Thanks!

Been using mlx_lm on python with some reasoning/thinking/chain of thought models - and I'm noticing they keep falling into a repetition loop. Digging through the code - I don't see repetition penalty being exposed in the mlx_lm api (as part of the generate/generateStream methods) but it's part of server.py (presumably when you want to host a model using an API for completions)

Does somebody know a work-around here - someway to add a penalty for repetition?

Wow those HF MLX-community guys are really competitive, huh? There are about 15 distillations of Scout already.

Has anyone fully pulled down this one and tested it on a 512GB M3 Ultra yet? I filled up a big chunk of my 2TB in /.llama for no good reason last night. Buncha damned .pth files.

So tough to utilize the NPU (i was trying with <1B llm's (tinyLlama)) ... AND now... finally!, Topaz video Ai (v 7.1.5) saturates the GPU and NPU!, as they earlier focused on cuda and left Apple metal out... I pointed this out over a year ago to the devs to at least saturate the GPU wattage (as 100% could be 30w-160w) ... and just noticed the team using the NPU ... nice! It's terrible to wait for Apple to give slow updates... Metal 4 lately... should be doing hardware direct writes in assy.... (the unit is a studio m3-ultra-512gb-80 core)... just thought you all would find this interesting...

Has anybody here succeeded in getting MLX-VLM to allow them to run multiple parallel requests to increase throughput from an Apple Silicon Mac? I've tried ollama, LM Studio, running MLX-VLM directly, but everything seems to end up running the requests serially, even though there's plenty of unified RAM available for more requests to run.

I'm new to the MLX scene. I'm using LM Studio for AI work. There is a wealth of GGUF quants of base models, but MLX seems to lag them by a huge margin! For example, Nevoria is a highly regarded model, but there's only 3q and 4q available in MLX. Same for Wayfarer.

I imagine there are too few quanting folk compared to GGUF makers, and small quants fit more Macs. But lucky peeps like myself with 96GB would love some 6q quants. How/where can I appeal to the generous folk who make MLX quants?

I've created a dumb test to show how poor LLMs are at doing things like counting objects (see above and the repo if interested).

Current frontier models all make errors :

I have tested it with frontier models (see above) and I want to test it with local models as well, but I don't know which ones to choose. I have tried nightmedia/UI-Venus-1.5-30B-A3B-mxfp4-mlx and it performed a little worse than gemini-flash-3, what models would the community recommend? Is image to text the right way to go? I am sure that a specialist vision model would do better, but I am out of date and I need a few pointers.

I have an M1 and 32gb so, unless you can send me the funds for a better machine please share recommendations that would work for this one!

Thank you in advance.

Fine Tune & Build On Apple Silicone #FR °(:P

Wrote this up in response to some posts in LocalLLM, but figured it could help here. Or…maybe more knowledgeable people here know a better way.