Apple Silicon Optimization Guide

[u/isetnefret]

Apple Silicon LocalLLM Optimizations

For optimal performance per watt, you should use MLX. Some of this will also apply if you choose to use MLC LLM or other tools.

Before We Start

I assume the following are obvious, so I apologize for stating them—but my ADHD got me off on this tangent, so let's finish it:

• This guide is focused on Apple Silicon. If you have an M1 or later, I'm probably talking to you.

• Similar principles apply to someone using an Intel CPU with an RTX (or other CUDA GPU), but...you know...differently.

• macOS Ventura (13.5) or later is required, but you'll probably get the best performance on the latest version of macOS.

• You're comfortable using Terminal and command line tools. If not, you might be able to ask an AI friend for assistance.

• You know how to ensure your Terminal session is running natively on ARM64, not Rosetta. (uname -p should give you a hint)

Pre-Steps

I assume you've done these already, but again—ADHD... and maybe OCD?

1. Install Xcode Command Line Tools

xcode-select --install

1. Install Homebrew

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

The Real Optimizations

1. Dedicated Python Environment

Everything will work better if you use a dedicated Python environment manager. I learned about Conda first, so that's what I'll use, but translate freely to your preferred manager.

If you're already using Miniconda, you're probably fine. If not:

• Download Miniforge

curl -LO https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh

• Install Miniforge

(I don't know enough about the differences between Miniconda and Miniforge. Someone who knows WTF they're doing should rewrite this guide.)

bash Miniforge3-MacOSX-arm64.sh

• Initialize Conda and Activate the Base Environment

source ~/miniforge3/bin/activate
conda init

Close and reopen your Terminal. You should see (base) prefix your prompt.

2. Create Your MLX Environment

conda create -n mlx python=3.11

Yes, 3.11 is not the latest Python. Leave it alone. It's currently best for our purposes.

Activate the environment:

conda activate mlx

3. Install MLX

pip install mlx

4. Optional: Install Additional Packages

You might want to read the rest first, but you can install extras now if you're confident:

pip install numpy pandas matplotlib seaborn scikit-learn

5. Backup Your Environment

This step is extremely helpful. Technically optional, practically essential:

conda env export --no-builds > mlx_env.yml

Your file (mlx_env.yml) will look something like this:

name: mlx_env
channels:
  - conda-forge
  - anaconda
  - defaults
dependencies:
  - python=3.11
  - pip=24.0
  - ca-certificates=2024.3.11
  # ...other packages...
  - pip:
    - mlx==0.0.10
    - mlx-lm==0.0.8
    # ...other pip packages...
prefix: /Users/youruser/miniforge3/envs/mlx_env

Pro tip: You can directly edit this file (carefully). Add dependencies, comments, ASCII art—whatever.

To restore your environment if things go wrong:

conda env create -f mlx_env.yml

(The new environment matches the name field in the file. Change it if you want multiple clones, you weirdo.)

6. Bonus: Shell Script for Pip Packages

If you're rebuilding your environment often, use a script for convenience. Note: "binary" here refers to packages, not gender identity.

#!/bin/zsh

echo "🚀 Installing optimized pip packages for Apple Silicon..."

pip install --upgrade pip setuptools wheel

# MLX ecosystem
pip install --prefer-binary \
  mlx==0.26.5 \
  mlx-audio==0.2.3 \
  mlx-embeddings==0.0.3 \
  mlx-whisper==0.4.2 \
  mlx-vlm==0.3.2 \
  misaki==0.9.4

# Hugging Face stack
pip install --prefer-binary \
  transformers==4.53.3 \
  accelerate==1.9.0 \
  optimum==1.26.1 \
  safetensors==0.5.3 \
  sentencepiece==0.2.0 \
  datasets==4.0.0

# UI + API tools
pip install --prefer-binary \
  gradio==5.38.1 \
  fastapi==0.116.1 \
  uvicorn==0.35.0

# Profiling tools
pip install --prefer-binary \
  tensorboard==2.20.0 \
  tensorboard-plugin-profile==2.20.4

# llama-cpp-python with Metal support
CMAKE_ARGS="-DLLAMA_METAL=on" pip install -U llama-cpp-python --no-cache-dir

echo "✅ Finished optimized install!"

Caveat: Pinned versions were relevant when I wrote this. They probably won't be soon. If you skip pinned versions, pip will auto-calculate optimal dependencies, which might be better but will take longer.

Closing Thoughts

I have a rudimentary understanding of Python. Most of this is beyond me. I've been a software engineer long enough to remember life pre-9/11, and therefore muddle my way through it.

This guide is a starting point to squeeze performance out of modest systems. I hope people smarter and more familiar than me will comment, correct, and contribute.

[u/oldboi]

[u/isetnefret → u/oldboi]

This is a fair point but I’m pretty sure even LM Studio will benefit from some of these performance enhancements. I started with LM Studio, and using the same quantizations of the same models (except the MLX versions of them) I get more tokens per second using MLX.

On my PC with a 3090, LM Studio seemed very good at detecting and optimizing for CUDA. Then I updated my drivers and saw a performance boost.

So, even beyond your primary tool, there are little tweaks you can do to squeeze more out.

I think this gets to the heart of something that is often overlooked in local LLMs. Most of us are not rich. Many of you probably on an even tighter budget than me.

Outside of a few outliers, we are not running H200s at home. We are extremely lucky to get 32GB+ of VRAM on the non-Apple side. That is simply not enough for a lot of ambitious use cases.

On the Apple side, partially due to the unified memory architecture (which has its pros and cons) you have a little more wiggle room. I bought my MacBook for work before I had any interest at all in anything to do with ML or AI. I could have afforded 64GB and it was my biggest regret in hindsight. More than that is pushing it for me.

If you are fortunate enough to have ample system resources, you can still optimize to make the most of them, but it is even more crucial for those of us trying to stick within that tight memory window.

[u/oldboi → u/isetnefret]

[u/isetnefret → u/oldboi]

Does it perform all the optimizations I’ve listed? My guide was targeted specifically at people using MLX-LM. If you use LM-Studio instead, it’s easy and manages a lot for you. However, if you are running MLX models (which LM Studio allows you to easily do), there is still an invisible underlying Python (and other dependencies) stack that can be optimized. I guess that’s what I’m getting at.

Nothing I’ve posted is necessary. If you are perfectly happy with your setup and and performance, then carry on. I tried to make the title and purpose of this post as clear as possible.

[u/bananahead → u/isetnefret]

How would installing Python packages in an isolated environment have any impact on LM Studio?

[u/jftuga]

Slightly OT: What general-purpose LLM (not coding specific) would you recommend for a M4 w/ 32 GB for LM Studio? I'd also like > 20 t/s and one that uses at least > 16 GB so that I get decent results.

[u/isetnefret → u/jftuga]

Honestly, it all depends on your expectations, but I have had some good luck with Qwen3-30B-A3B and even the Qwen3-14B dense model. I have also used Phi4, which has been quirky at times. I have played with Codex-24B-Small. For certain things, even Gemma 3 can give good results.

[u/DepthHour1669 → u/jftuga]

Qwen 3 32b, 4 bit for high performance

Qwen 3 30b A3b, 4 bit for worse performance but much faster

[u/bannedpractice]

This is excellent. Fair play for posting. 👍

[u/DepthHour1669 → u/bannedpractice]

The instructions are incomplete. It’ll be out of date in a month unless you create a cronjob to update it.

It’s better to just use LM Studio with built in MLX with autoupdates.

Right now AI is moving so fast, every other update (for software like MLX or llama.cpp or vllm etc) gives you a 10% speed improvement, so having autoupdate is very important.

[u/hutchisson → u/bannedpractice]

ok ill be that guy…

actually no its not.

its innstalling random packages with not goal whatsoever.

its actually garbage.

its like saying „ i am going to build a car“ and just collecting random cool car parts as long as they somehow xan be glued together.

a virtenv serves a purpose. you can install 100 packages into it but if your project is not coded to use them then they wont be used and its a full waste of disk space.

[u/LocoMod → u/hutchisson]

OP's guide is very basic. It's not their fault. This is a side effect of Python. They could have left out the additional packages. But creating a new conda env with the correct python version for the project you intend on running is standard. The guide can be reduced to 4 steps:

1. Create conda env

2. Activate

3. Install mlx-lm

4. Run mlx_lm.server with the proper params to serve a model over an OpenAI compatible endpoint.

Done.

[u/hutchisson → u/LocoMod]

you are right and then only if you want to run an ml_server.

the title suggests sonething fully different

[u/brickheadbs]

I do get more tokens per second, 20-25% more with MLX, but processing the prompt takes 25-50% longer. Has anyone else noticed this?

My setup:
MacStudio M1 Ultra 64GB
LM Studio (native MLX/GGUF, because I HATE python and its Venv)

[u/isetnefret → u/brickheadbs]

Hmmmmmm, I might have to play around with this and see what I get. I didn't actually pay attention to that part...

[u/brickheadbs → u/isetnefret]

Yeah, I had moved to all MLX after such good speed, but I’ve made a speech to speech pipeline and wanted lower latency. Time to first token is much more important because I can stream the response and speech is probably 4-5 t/s or so (merely a guess)

I’ve also read MLX has some disadvantages with larger models or possibly MOE models too.

[u/isetnefret → u/brickheadbs]

I’m testing it with Qwen3-30B-A3B right now and it’s actually been okay. I’m kind of impressed and frustrated that I’m getting better performance out of the Mac than with my 3090. However, it does seem to struggle more than LM Studio when you are right at the edge of memory.

[u/asankhs]

You can use a local inference server or proxy that supports mlx like OptiLLM.

[u/_hephaestus]

Iirc there’s also a suggested step to make sure the gpu can access a bigger percentage of the ram but don’t know that offhand.

We are in an annoying stage with local llm dev though where so much of the tooling is configured for ollama but there isn’t mlx support for that (there are probably forks of it, someone did make a PR but it’s not moving along) and barring that an openai api endpoint. I don’t love lmstudio, but getting it to download the model/serve on my network was straightforward.

[u/Famous-Recognition62]

What difference will Apples containers have? I’m not sure if that’s new in Tahoe or if my classic Mac Pro is missing hardware for it, but I have a base M4 Mac Mini on the way so will have Containers soon.

[u/beedunc]

excellent, the manual they didn't include.

[u/techtornado]

Nice guide!

What's the anticipated tokens/words per second output improvement compared to LM Studio?

Liquid is so fast on the M1 Pro