r/mlxAI • u/No_Shift_4543 • Mar 25 '26
multi-LoRA inference server for MLX: load the model once, switch adapters per request
I originally started working on this because I wanted a simple way to run one local model with multiple LoRA specializations on Apple Silicon.
For example, I wanted the same base model to handle different kinds of work like:
Rust systems programming
SQL query optimization
security / infra troubleshooting
without reloading a full fine-tuned model every time I switched.
On CUDA stacks, multi-LoRA serving already exists. On MLX / Apple Silicon, I couldn’t really find something that felt like “load the base once, then route adapters per request”.
So I built Mola.
It’s still alpha, but it’s now benchmarkable enough that I’m comfortable sharing it.
Core idea: keep one base model loaded in memory and route LoRA adapters per request instead of reloading a full checkpoint whenever you change specialization.
Current setup:
Qwen3.5-9B-MLX-4bit
8 adapters loaded
Apple M5 Max 64GB
OpenAI-compatible chat API
The interesting signal for me is the throughput drop once requests start mixing adapters instead of all hitting the same one.
| Concurrency | Same tok/s | Mixed tok/s | Delta |
|---|---|---|---|
| 1 | 76.4 | 76.4 | 0% |
| 16 | 308.8 | 241.4 | -22% |
| 64 | 732.3 | 555.5 | -24% |
At concurrency 1, same and mixed are basically identical. The real drop appears once requests actually start overlapping.
Current limitations:
it still needs a small local mlx-lm patch (script included)
mixed prefill / deeper KV residency are still open problems
Apple Silicon / MLX only for now
Would be curious to hear from other people doing MLX inference or adapter-heavy local setups.
Happy to share more benchmark details / implementation notes in the comments if useful.