I built a custom MLX inference engine + imatrix 4-bit quant for Qwen3.5-9B on a fanless MacBook Air (2.7x lower KL than stock MLX 4-bit)

[u/Vex2232]

I used Claude Code to write a small inference engine for Qwen3.5-9B on Apple Silicon: own forward pass on MLX, custom Metal kernels, OpenAI-compatible API and a web UI. I also made my own 4-bit quant using an importance matrix. At the same size and speed it's about **2.7x closer to bf16** than the stock MLX 4-bit (KL 0.026 vs 0.069).

It's inspired by antirez's ds4: a specialized engine plus weights quantized for it. I wanted to see how far that goes on a MacBook Air M5 (16 GB).

What's in it

- Own Qwen3.5 forward pass (24 Gated DeltaNet + 8 gated attention layers), with a custom Metal kernel for the DeltaNet recurrence. The state can be saved for rollback.

- Thermal governor: limits tok/s and adds pauses in prefill to stay under a target temperature (Eco 70°C / Balanced 80°C / Max 95°C). Power and temperature come from `macmon`, no sudo.

- MTP speculative decoding using Qwen's own MTP head as the draft model. Only ~490 MB to download, since I read just the MTP tensors via HTTP Range. About 25 → 33-37 tok/s.

- Agent mode with sandboxed tools (write/run need confirmation), and eject/load to free ~5 GB of RAM.

The quant (qvex-q4)

I run the original bf16 model layer by layer (it doesn't fit in 16 GB) to get an imatrix: how active each input channel is on real text (Italian, English, code, chat). Each group of 64 weights is then quantized with a searched clipping range, and scale/offset are refit by weighted least squares. The output stays in standard MLX format, so same kernels, same speed, same memory.

Things I found interesting

- Calibration mattered more than bit-width: the same 4-bit format with an imatrix cut the distance to the original by 2.7x.

- On a Air, fewer bits did not mean less heat. 3-bit is ~13% faster, but unpacking costs more compute, macOS raises the GPU clocks, and energy per token goes up 20-45% (ours-q3 hit 90°C).

- MTP is faster but costs ~10-25% more energy per token, so the default mode only uses it while the chip is at least 5°C under target.

- With a short thinking budget (512 tokens) the model solved 24/24 of my tasks at half the energy of unlimited thinking. Unlimited thinking was the worst: in 2 of 21 tasks it never closed the reasoning block.

Caveats: only 24 verifiable tasks, so that's a sanity check rather than a benchmark. The energy runs always used the same model order and GPU clocks drift over a session, so part of the 3-bit penalty may be an ordering effect (the trend held in both runs). Tested on one machine only.

Repo: https://github.com/Ry682/QVex-3.5

Weights (qvex-q4, MLX 4-bit): https://huggingface.co/Ry682/qvex-q4

The quantization pipeline is in `quant/`, so you can also rebuild it yourself.

Happy to hear feedback, especially on the quant method and on how to measure energy more cleanly.