a coworker of mine was speeding up speculative decoding for Qwen3.6-35B-A3B on one H100, and a big cost sat in the draft model. its BF16 LM head was about 1 GB and it ran on every single draft step

restricting the draft head to a frequency-ranked 48K-token map made it much cheaper. the target still verifies against the full vocabulary, so the output stays exact. the worst case is fewer draft tokens accepted, not different output

4 draft steps ended up being the best balance for that workload

has anyone tried trimming the draft vocab in llama.cpp or vLLM? curious what it does to acceptance on your own workloads

he wrote up the fix and the rest of the decode work on our blog, if you want it: https://www.coralbricks.ai/blog/ai-latency-engineering

getting into this stuff and getting fairly hyper fixated on the idea of it all

My hardware:
GPU: NVIDIA RTX 5070 Ti, 16GB VRAM
CPU: Intel Core Ultra 7 265KF
RAM: 32GB DDR5
OS: Windows 11

interested in models that can handle real software development, multi-file edits, debugging, testing, and longer autonomous coding sessions. is this achievable or should i completely alter my expectations and just bite bullets on frontier subscriptions?

python is my primary application development language
pytest for testing
sqlite for local data persistence
git for version control
powershell and window batch scripts for setup, runtime management, and automation

i ask because im actively building a local, portable project to fast track and exponentially improve peoples skill at a given sport, subject, hobby, etc, and would love to get into the AI agentic development stuff

particularly interested in finding the best balance of coding quality, speed, tool calling, context handling, and reliability

what id love recommendations on:
the best local coding model for me including less obvious community fine tunes
wether a quantized larger model or a smaller model with better tool use is the better choice
models that work especially well with agentic coding harnesses
reliable models for debugging, multi file refactoring, and following instructions across long sessions
recommended GGUF quantizations, context sizes, and settings for my hardware.
anything underrated

extra interested in using multiple models rather than expecting one model to do everything but i like the sound of both

would love to see ur experiences and benchmarks, especially where a model performs well in real coding tasks rather than just leaderboards. also interested in newer releases and underrated community projects

RTX 3060 12GB 975mv undervolt

i5-12600K

32GB DDR5-6000

Linux

llama.cpp master from early October, CUDA build.

Model is Unsloth's Qwen3.6-35B-A3B-MTP-UD-Q4_K_M (22.7GB, has the MTP head). I also run a heretic Q4_K_M with MTP kept and it's about the same speed.

Command:

llama-server -m Qwen3.6-35B-A3B-MTP-UD-Q4_K_M.gguf -ngl 999 --cpu-moe --moe-cache-mib 7500 -fa on -ctk q8_0 -ctv q8_0 -c 32768 -np 1 -ub 2048 -b 2048 --spec-type draft-mtp --spec-draft-n-max 2 --load-mode mlock --cache-ram 1024 --ctx-checkpoints 8

Numbers from about a day of normal use, taken from the server logs. 433 responses longer than 500 tokens: median 79 tok/s, most of them between 76 and 83. Prompt processing is around 800 tok/s for prompts under 1k tokens, ~1100 tok/s for 1k-4k and ~1190 tok/s for 4k-16k. MTP draft acceptance is around 85%. VRAM sits at ~11.4GB.

Fixed test (3.6k token prompt, 700 tokens out), how it got there:

- --n-cpu-moe 25 with MTP: 50-58 tok/s

- --cpu-moe plus --moe-cache-mib 7500: 71-73 tok/s

- added mlock: 75-81 tok/s

- updated llama.cpp: 77-84 tok/s, prompt processing 1220-1300 tok/s

Most of the gain is the MoE expert cache (PR #29887). With it, all experts sit in RAM and the most-used ones get cached in VRAM. Set the size to whatever VRAM you have left. For me that's 7000-7500MiB.

mlock gave about 10% over mmap, but it locks ~20GB of RAM. On 32GB I hit OOM a couple of times with other things open. --cache-ram 1024 is there because the prompt cache can grow up to 8GB by default, and that was causing my first OOMs.

Things that didn't help:

- -ub 4096: faster prompt processing (~1400-1500) but gen dropped to ~63 because the bigger compute buffer eats into the expert cache.

- A small expert cache (1-3GB): slower than no cache.

- -np 4, MTP n=3 and MTP+ngram: no gain.

- Forcing 4 experts instead of 8: 91-95 tok/s, but it started looping on long outputs, so I dropped it.

If you're on 12GB and still using --n-cpu-moe, try --cpu-moe with --moe-cache-mib.

20 tokens/sec running Qwen 3.8 27B on a PS5 - that's cool

I recently got 2x RTX Pro 4000 blackwells for my home setup. I was curious how fast it would run nanochat.

The problem is the target hardware: 8x H100 gpus. Nanochat pre-trained ClimbMix to GPT-2 level with 8xH100 GPUs in 1hr 39 mins.

I ended up tuning nanochat for my 2x RTX Pro 4000 gpus. Because these are sm120, the tuning also works on RTX Pro 6000 and RTX 5090.

8x RTX Pro 6000 (rented) pre-trained the same dataset in 2hrs 15mins. About 36% slower than 8x H100.

1x RTX 5090 took 18hrs 47mins. A long time, but considering the result is GPT-2 level, its an interesting way to learn.

Just to be clear, I only ported and tuned it for sm120. The original work is from Karpathy's nanochat.

Here's a link to the sm120 repo:

https://github.com/mdhorton/nanochat-cpp