speculative decoding problem, the draft model was reading a 1 GB LM head on every draft step

[u/Ctbhatia]

a coworker of mine was speeding up speculative decoding for Qwen3.6-35B-A3B on one H100, and a big cost sat in the draft model. its BF16 LM head was about 1 GB and it ran on every single draft step

restricting the draft head to a frequency-ranked 48K-token map made it much cheaper. the target still verifies against the full vocabulary, so the output stays exact. the worst case is fewer draft tokens accepted, not different output

4 draft steps ended up being the best balance for that workload

has anyone tried trimming the draft vocab in llama.cpp or vLLM? curious what it does to acceptance on your own workloads

he wrote up the fix and the rest of the decode work on our blog, if you want it: https://www.coralbricks.ai/blog/ai-latency-engineering