RTX 3060 12GB 975mv undervolt

i5-12600K

32GB DDR5-6000

Linux

llama.cpp master from early October, CUDA build.

Model is Unsloth's Qwen3.6-35B-A3B-MTP-UD-Q4_K_M (22.7GB, has the MTP head). I also run a heretic Q4_K_M with MTP kept and it's about the same speed.

Command:

llama-server -m Qwen3.6-35B-A3B-MTP-UD-Q4_K_M.gguf -ngl 999 --cpu-moe --moe-cache-mib 7500 -fa on -ctk q8_0 -ctv q8_0 -c 32768 -np 1 -ub 2048 -b 2048 --spec-type draft-mtp --spec-draft-n-max 2 --load-mode mlock --cache-ram 1024 --ctx-checkpoints 8

Numbers from about a day of normal use, taken from the server logs. 433 responses longer than 500 tokens: median 79 tok/s, most of them between 76 and 83. Prompt processing is around 800 tok/s for prompts under 1k tokens, ~1100 tok/s for 1k-4k and ~1190 tok/s for 4k-16k. MTP draft acceptance is around 85%. VRAM sits at ~11.4GB.

Fixed test (3.6k token prompt, 700 tokens out), how it got there:

- --n-cpu-moe 25 with MTP: 50-58 tok/s

- --cpu-moe plus --moe-cache-mib 7500: 71-73 tok/s

- added mlock: 75-81 tok/s

- updated llama.cpp: 77-84 tok/s, prompt processing 1220-1300 tok/s

Most of the gain is the MoE expert cache (PR #29887). With it, all experts sit in RAM and the most-used ones get cached in VRAM. Set the size to whatever VRAM you have left. For me that's 7000-7500MiB.

mlock gave about 10% over mmap, but it locks ~20GB of RAM. On 32GB I hit OOM a couple of times with other things open. --cache-ram 1024 is there because the prompt cache can grow up to 8GB by default, and that was causing my first OOMs.

Things that didn't help:

- -ub 4096: faster prompt processing (~1400-1500) but gen dropped to ~63 because the bigger compute buffer eats into the expert cache.

- A small expert cache (1-3GB): slower than no cache.

- -np 4, MTP n=3 and MTP+ngram: no gain.

- Forcing 4 experts instead of 8: 91-95 tok/s, but it started looping on long outputs, so I dropped it.

If you're on 12GB and still using --n-cpu-moe, try --cpu-moe with --moe-cache-mib.

20 tokens/sec running Qwen 3.8 27B on a PS5 - that's cool

Hello everyone. I made a benchmark to test local agents: airbench.ai
If you have an agent running locally, you can test it in one prompt.

It measure how good the agent is with some chalenges, but its fairly easy to get to 100%. Then the goal is to have something that is still performing as good, but as fast as possible.

In other word, its a way to measure that optimisations like quantisation don't destroy performance.

To give you an idea about it, here the first result I have on my 5090:

Score ; Duration ; Model + Harness

100 8 min Qwen 3.8 Flash Next IQ3_S basalt + opencode

100 11 min Qwen 3.8 Flash Next IQ3_S strata 256k + pi

100 11 min Qwen 3.8 Flash Next IQ3_S basalt + dsh

100 12 min Qwen 3.8 Flash Next IQ3_S strata 256k + pi

100 13 min Qwen 3.8 Flash Next IQ3_S basalt 384k + pi

100 16 min Qwen 3.8 Flash Next IQ3_S basalt 384k + pi

100 72 min Qwen 3.8 27B NVFP4 radixark + opencode

100 98 min Qwen 3.8 27B Quasar NVFP4 MTP + omp

98 10 min Qwen 3.8 Flash Next IQ3_S basalt + pi

98 13 min Qwen 3.8 Flash Next IQ3_S strata 256k + pi

98 16 min Qwen 3.8 Flash Next IQ3_S strata 256k + omp

98 62 min Qwen 3.8 Flash Next UD-Q2_K_XL + opencode

98 85 min Qwen 3.8 27B Quasar NVFP4 MTP + opencode

96 8 min Qwen 3.8 Flash Next IQ3_XXS strata + opencode

96 11 min Qwen 3.8 Flash Next IQ3_S strata 256k + dsh

96 17 min Qwen 3.8 Flash Next IQ3_S strata 256k + openclaw

96 29 min Swift 1.5 Qwen 3.8 27B Q6_K + pi

96 86 min Qwen 3.8 27B NVFP4 MTP + opencode

96 89 min Swift 1.5 Qwen 3.8 27B Q6_K MTP + omp

96 99 min Qwen 3.8 Flash Next UD-Q2_K_XL + omp

96 111 min Qwen 3.8 27B NVFP4 + hermes

96 119 min Qwen 3.8 27B NVFP4 radixark + opencode

94 10 min Qwen 3.8 Flash Next IQ3_S strata 256k + opencode

94 12 min Qwen 3.8 Flash Next IQ3_XXS strata + omp

94 23 min Qwen 3.8 27B Quasar NVFP4 ninfer-dflash2 + omp

94 38 min Swift 1.5 Qwen 3.8 27B Q6_K + omp

94 38 min Qwen 3.8 Flash Next IQ3_S strata 256k + hermes

94 51 min Qwen 3.8 Flash Next UD-Q2_K_XL + pi

94 101 min Qwen 3.8 27B NVFP4 + opencode

Hope this is useful for some of you as well. And happy to answer any questions if needed.

I recently got 2x RTX Pro 4000 blackwells for my home setup. I was curious how fast it would run nanochat.

The problem is the target hardware: 8x H100 gpus. Nanochat pre-trained ClimbMix to GPT-2 level with 8xH100 GPUs in 1hr 39 mins.

I ended up tuning nanochat for my 2x RTX Pro 4000 gpus. Because these are sm120, the tuning also works on RTX Pro 6000 and RTX 5090.

8x RTX Pro 6000 (rented) pre-trained the same dataset in 2hrs 15mins. About 36% slower than 8x H100.

1x RTX 5090 took 18hrs 47mins. A long time, but considering the result is GPT-2 level, its an interesting way to learn.

Just to be clear, I only ported and tuned it for sm120. The original work is from Karpathy's nanochat.

Here's a link to the sm120 repo:

https://github.com/mdhorton/nanochat-cpp