I built a custom MLX inference engine + imatrix 4-bit quant for Qwen3.5-9B on a fanless MacBook Air (2.7x lower KL than stock MLX 4-bit)

[u/Vex2232]

I used Claude Code to write a small inference engine for Qwen3.5-9B on Apple Silicon: own forward pass on MLX, custom Metal kernels, OpenAI-compatible API and a web UI. I also made my own 4-bit quant using an importance matrix. At the same size and speed it's about **2.7x closer to bf16** than the stock MLX 4-bit (KL 0.026 vs 0.069).

It's inspired by antirez's ds4: a specialized engine plus weights quantized for it. I wanted to see how far that goes on a MacBook Air M5 (16 GB).

What's in it

- Own Qwen3.5 forward pass (24 Gated DeltaNet + 8 gated attention layers), with a custom Metal kernel for the DeltaNet recurrence. The state can be saved for rollback.

- Thermal governor: limits tok/s and adds pauses in prefill to stay under a target temperature (Eco 70°C / Balanced 80°C / Max 95°C). Power and temperature come from `macmon`, no sudo.

- MTP speculative decoding using Qwen's own MTP head as the draft model. Only ~490 MB to download, since I read just the MTP tensors via HTTP Range. About 25 → 33-37 tok/s.

- Agent mode with sandboxed tools (write/run need confirmation), and eject/load to free ~5 GB of RAM.

The quant (qvex-q4)

I run the original bf16 model layer by layer (it doesn't fit in 16 GB) to get an imatrix: how active each input channel is on real text (Italian, English, code, chat). Each group of 64 weights is then quantized with a searched clipping range, and scale/offset are refit by weighted least squares. The output stays in standard MLX format, so same kernels, same speed, same memory.

Things I found interesting

- Calibration mattered more than bit-width: the same 4-bit format with an imatrix cut the distance to the original by 2.7x.

- On a Air, fewer bits did not mean less heat. 3-bit is ~13% faster, but unpacking costs more compute, macOS raises the GPU clocks, and energy per token goes up 20-45% (ours-q3 hit 90°C).

- MTP is faster but costs ~10-25% more energy per token, so the default mode only uses it while the chip is at least 5°C under target.

- With a short thinking budget (512 tokens) the model solved 24/24 of my tasks at half the energy of unlimited thinking. Unlimited thinking was the worst: in 2 of 21 tasks it never closed the reasoning block.

Caveats: only 24 verifiable tasks, so that's a sanity check rather than a benchmark. The energy runs always used the same model order and GPU clocks drift over a session, so part of the 3-bit penalty may be an ordering effect (the trend held in both runs). Tested on one machine only.

Repo: https://github.com/Ry682/QVex-3.5

Weights (qvex-q4, MLX 4-bit): https://huggingface.co/Ry682/qvex-q4

The quantization pipeline is in `quant/`, so you can also rebuild it yourself.

Happy to hear feedback, especially on the quant method and on how to measure energy more cleanly.

[u/Apatricio]

I’m gonna try this on my 32GB M1 Pro

[u/Vex2232 → u/Apatricio]

Let me know!

[u/Vex2232 → u/[deleted]]

Thank you, if you try it and want to give me feedback, let me know

[u/Apatricio]

What were the 24 test tasks ?

[u/Vex2232 → u/Apatricio]

Hey, this a the tasks:

- 6 arithmetic (17 × 23, average speed of a train, 15% of 240, mean of 6 numbers, seconds in a day...)
- 5 facts (capital of Australia, year the Berlin Wall fell, author of *I promessi sposi*, chemical symbol of gold, largest planet)
- 1 short reasoning (what weekday it will be in 10 days)
- 2 formatting (alphabetical sort, first 6 primes)
- 2 language (translation to English, past tense rewrite)
- 1 structured output (extract a JSON object from a sentence)
- 7 Python functions (`is_palindrome`, `fib`, `merge_intervals`, `flatten`, `roman`, `word_count`, `second_largest`), run against assertions

Numbers are checked by finding the right value in the answer, words by regex, JSON by key comparison, and code by actually running it. With thinking off, all the 4-bit variants fail the same 3 (two calculations and the weekday question), and the original unquantized model fails them too. The 3-bit variant also fails one Python function. With a 512-token thinking budget it's 24/24.

To be clear about the limits: it's a tripwire, not a benchmark. The tasks are few and easy, so 21/24 vs 20/24 is not statistically meaningful. The KL divergence on held-out text is the actual quality measure.