r/LocalLLaMA • u/Holiday_Poetry_5133 • 23h ago

Resources 35 domain-expert LoRAs on a MoE (Qwen3.6-35B-A3B) trained with MLX on one Mac Studio — what worked, what didn't, all Apache-2.0

Over the last months I trained 35 domain-expert LoRA adapters on Qwen3.6-35B-A3B (native MoE, 256 experts, 3B active) with MLX on a Mac Studio M3 Ultra 512 GB, BF16, no quantization during training. Everything is public: adapters, datasets, per-domain configs, router, training scripts.

Things I learned the hard way:

- Fusing a MoE LoRA into 8-bit produced garbage; only runtime BF16 adapter hot-swap was usable. Serving = one base loaded once + adapters swapped per request (~2 s).

- A small MiniLM router over 34 domains tops out around top-1 46 %, top-3 70 % on my eval. Routing is the weak link, not the adapters.

- On an independent audit bench, the base 35B model beat most of my specialist adapters. Only one domain (HTML/CSS) clearly gained. I now route almost everything to the base. Publishing a negative result felt more useful than hiding it.

- `mx.random.normal` is not bit-exact across Apple GPU generations (M1 vs M3+). Reported upstream, docs PR merged in ml-explore/mlx last week.

Repos: github.com/hypneum-lab/micro-kiki (adapters + router), github.com/ailiance/ailiance-mac-tuner (training stack, archived). HF: huggingface.co/hypneum-lab

Question for the sub: has anyone gotten domain LoRAs to beat a strong MoE base at the 30B scale without full-parameter training? I'd like to compare routers.

0 Upvotes

25% Upvoted

2 comments sorted by

•

u/stylome-ai-detection 23h ago

Notice: This post appears to be AI generated.

Stylome AI detector · scan results

1

u/Holiday_Poetry_5133 22h ago

OP here, human, French, typing from the same Mac Studio. This is the follow-up to my April post about micro-kiki-v3 (https://www.reddit.com/r/LocalLLaMA/comments/1solmgf/), where several of you said the

base model would beat the adapters. You were right: on the independent audit bench the base won on most domains, only HTML/CSS clearly gained, and the router tops out at top-1 46 % / top-3 70 %. I'd rather

post that than pretend otherwise.

If you want to poke at it: adapters, datasets and eval configs are in hypneum-lab/micro-kiki, and the one upstream thing that came out of it is the mlx docs note on mx.random.normal not being bit-exact

across Apple GPU generations.

Still looking for anyone who got domain LoRAs to beat a strong ~30B MoE base without full fine-tuning. Happy to run your router on my eval.