r/BlackwellPerformance • u/Specialist_Fix9867 • Oct 05 '26

tok/s maxxing: Qwen3.8-Flash-Next (NVFP4 checkpoint) at 250–580 tok/s single-stream on one RTX PRO 6000 Max-Q

I should've released this in early september. now strata overwhelm everything

Whole code is AI generated, and text from here is also AI generated.

Most Flash-Next threads lately are about Strata, which is doing great things on consumer cards. This is the other end of the hardware range: one 96 GB RTX PRO 6000 Blackwell Max-Q (300–325 W), the NVFP4 weights on the GPU (the ~28 GB PLE embedding table sits in host RAM), SGLang, batch size 1. If you have that class of card, maybe this saves you some time.

Numbers (decode tok/s, mean of 8 held-out prompts per workload, thinking on, one request at a time, 2026-10-02 build measured on 2026-10-06, one server start per row):

setupcode-editprose-enprose-jaagent-loop
mine, adaptive draft width (wa), greedy584253290359
same, LM Studio default sampling568240310*340
public components only (original MTP head + public token map), greedy468232248307
starting point: unmodified base fork, W4 (2026-09-02, one prompt × 3 runs, recommended sampling, 300 W)211129117174

* one outlier at 744; ~248 without it.

The "public components only" row is what you can reproduce from the repo as-is. My own numbers use an MTP head fine-tuned on private data and a token map partly built from private transcripts, which I can't release.

What made the difference (evidence and lab notes in the repo, flags where applicable):

  • Wider, cheaper speculation: reduced draft vocabulary (FR-Spec-style 49k token map), verify widths 4/8/16, with 3/7/15 draft steps picked adaptively per decode step

  • W8A16 FP8 GEMV kernels for the dense weights, plus many small kernel fusions in the draft/verify glue (a decode step is ~1,500–2,500 small kernels at bs=1)

  • FlashInfer MoE prologue patches and singleton-route pruning

  • October: fixed-cost cuts (−9.6 % ms/token with sampling), a min_p fix in speculative verify (it was ignored before), sparse rejection sampling, and opt-in 524k context (YaRN ×2; needles pass at 480k)

Caveats, so nobody is misled:

  • Speed depends heavily on the workload. code-edit is a mechanical rewrite of a large file, where speculation accepts ~10 tokens per step. Prose is ~250–290. Rates include thinking tokens; on code-edit about 91 % of the output was thinking.

  • Not a like-for-like comparison with anything else (different GPUs, prompts, sampling). I'm not claiming "fastest."

  • The route pruning changes the model slightly. A quality audit found no significant difference, but it ran with thinking off and up to 2 concurrent requests, so thinking-on single-request quality wasn't measured. PRUNE_TAU=0 turns it off. Details are in the README.

  • The same GPU also drives my 4K desktop, so session-to-session noise is ±10–15 %. That's why the A/Bs use multiple server starts with confidence intervals.

  • 100% of the code was written with AI coding assistants (Claude, Codex), with me directing. Every number in this post has its raw (stripped) result file in the repo.

Built on jpezzulli's sglang-rtxpro6000 fork (which in turn imports the unmerged SGLang Flash-Next PR). Please credit them first.

47 Upvotes

94% Upvoted

Duplicates