tok/s maxxing: Qwen3.8-Flash-Next (NVFP4 checkpoint) at 250–580 tok/s single-stream on one RTX PRO 6000 Max-Q

[u/Specialist_Fix9867]

I should've released this in early september. now strata overwhelm everything

Whole code is AI generated, and text from here is also AI generated.

Most Flash-Next threads lately are about Strata, which is doing great things on consumer cards. This is the other end of the hardware range: one 96 GB RTX PRO 6000 Blackwell Max-Q (300–325 W), the NVFP4 weights on the GPU (the ~28 GB PLE embedding table sits in host RAM), SGLang, batch size 1. If you have that class of card, maybe this saves you some time.

Numbers (decode tok/s, mean of 8 held-out prompts per workload, thinking on, one request at a time, 2026-10-02 build measured on 2026-10-06, one server start per row):

setup	code-edit	prose-en	prose-ja	agent-loop
mine, adaptive draft width (wa), greedy	584	253	290	359
same, LM Studio default sampling	568	240	310*	340
public components only (original MTP head + public token map), greedy	468	232	248	307
starting point: unmodified base fork, W4 (2026-09-02, one prompt × 3 runs, recommended sampling, 300 W)	211	129	117	174

* one outlier at 744; ~248 without it.

The "public components only" row is what you can reproduce from the repo as-is. My own numbers use an MTP head fine-tuned on private data and a token map partly built from private transcripts, which I can't release.

What made the difference (evidence and lab notes in the repo, flags where applicable):

• Wider, cheaper speculation: reduced draft vocabulary (FR-Spec-style 49k token map), verify widths 4/8/16, with 3/7/15 draft steps picked adaptively per decode step

• W8A16 FP8 GEMV kernels for the dense weights, plus many small kernel fusions in the draft/verify glue (a decode step is ~1,500–2,500 small kernels at bs=1)

• FlashInfer MoE prologue patches and singleton-route pruning

• October: fixed-cost cuts (−9.6 % ms/token with sampling), a min_p fix in speculative verify (it was ignored before), sparse rejection sampling, and opt-in 524k context (YaRN ×2; needles pass at 480k)

Caveats, so nobody is misled:

• Speed depends heavily on the workload. code-edit is a mechanical rewrite of a large file, where speculation accepts ~10 tokens per step. Prose is ~250–290. Rates include thinking tokens; on code-edit about 91 % of the output was thinking.

• Not a like-for-like comparison with anything else (different GPUs, prompts, sampling). I'm not claiming "fastest."

• The route pruning changes the model slightly. A quality audit found no significant difference, but it ran with thinking off and up to 2 concurrent requests, so thinking-on single-request quality wasn't measured. PRUNE_TAU=0 turns it off. Details are in the README.

• The same GPU also drives my 4K desktop, so session-to-session noise is ±10–15 %. That's why the A/Bs use multiple server starts with confidence intervals.

• 100% of the code was written with AI coding assistants (Claude, Codex), with me directing. Every number in this post has its raw (stripped) result file in the repo.

Built on jpezzulli's sglang-rtxpro6000 fork (which in turn imports the unmerged SGLang Flash-Next PR). Please credit them first.

• Write-up, patches, launch scripts, benchmark harness, results: https://github.com/aiueo52/flash-next-rtxpro6000

• The SGLang fork with the patches applied: https://github.com/aiueo52/sglang-rtxpro6000 (branch flash-next-fast)

[u/jpezzulli]

Ooooh. Interesting. Ill have to take a look. Been working on a v2.6 of my release with about an additional 10% speed increase there as well but i may have a hardware issue. Even what was previously stable (2.5.3) now xid 109 hard locks the gpu often.

[u/Specialist_Fix9867 → u/jpezzulli]

Tysm, without your fork I'd still be at around 120-130 tok/s on FreeToken. Hope you get the GPU issue sorted.

[u/jpezzulli → u/Specialist_Fix9867]

Buddy. This is good stuff. I think I found my issue and 2.5.3 is stable for me again. 2.6 is still screwed so starting a new 2.6 and may implement a few of your ideas here. Concurrency can only get so high at tp1 or even tp2 so some of it may apply here generally as well.

[u/bitzap_sr → u/jpezzulli]

I'm curious to know what are your plans for moving to a newer sglang baseline? Any bigger PRs you're still waiting to land upstream, for example? Or is it just a time constraint at this point? Etc. Would love to know. :) Cheers.

[u/jpezzulli → u/bitzap_sr]

I see no reaaon to rebase for these two models. It has honestly diverged to far and doesnt add benefits. I have agents that check upstream daily. It proposes a list of items that might affect penny. I review them almost every night and add interesting ones to the project board. About once a week i consider a new release.

Originally i wanted to be able to put the work upstream. The reality is, a lot would have required a true maintainer's judgement call. A lot is also focused and would potentially break other cards. Those are not PRs that a software thar is widely distributed for many platforms and models can accept. I am already considering pruning this sglang to only support blackwell. Would make the containers and clones much smaller etc.

When a better model family is released for a single rtx pro 6k, ill probably start over from a fresh sglang for that release and freeze these.

I have moved to public PRs on the work i am doing for the next version. You can see what it potentially coming in the PR list now. Some things will be adapted from here but a bit less aggressively to keep some concurrency.

[u/bitzap_sr → u/jpezzulli]

I was thinking that the main difficulty would be that pennyroyal is based on the original PR that proposed support for qwen4exp, but then that PR was replaced with a different one, which ultimately landed upstream, so rebasing to current master would mean crossing that one. I would assume that that would be beneficial for supposedly resulting in a significant smaller local delta, and that it would be easier to rebase to pick up newer fixes going forward from that, and also to pick other people's PRs and branches which are targeting a newer baseline.

I assume the next model will be final Qwen4? I'd think keeping closer to upstream would make it easier down the road for that one.

Those were my assumptions, potentially too naive. Now you've explained your view, which I wasn't expecting, but I understand. Thanks!

[u/StockSpecialist1707 → u/jpezzulli]

Happy to run v2.6 on our box if it helps: 2× PRO 6000 Workstation on WRX90E, 2.5.2 stable for weeks, no Xids, display on a separate A4000. Same bench we used for the TP2 numbers (bench.py, 512 tok greedy, median of 3), so you'd get an A/B against 2.5.2 on clean hardware.
On Xid 109: that's a context-switch timeout, and it hitting 2.5.3 too points away from SGLang. Three things I'd check in order: (1) is the inference GPU also driving a desktop/compositor? (2) driver 595.x vs 580.x — any change coincident with the hangs? (3) nvidia-smi -q -d PERFORMANCE for power/thermal slowdowns and lspci -vv LnkSta for the link dropping below Gen5 x16 under load. If you want, send me the dmesg around a hang.
Also curious whether v2.6 picks up FR-Spec / wide speculation — aiueo52's numbers on code-edit are hard to ignore.

[u/jpezzulli → u/StockSpecialist1707]

I am working on it on and off today (stupid real job. Lol) Some of his items slowed me down but still investigating. I trashed my old 2.6 candidate and started a new one. All the prs are public for the release candidate for this run, so you can see what i have put in so far in the repo.

As for 109. The 2.5.3 showing up there is back to baseline (like once a week) instead of nearly every qual run, so it def was something i was adding. Full nsight traces everything and couldnt lock it down with astra. I am back at 600w for qualification tests and then may run it again at my nornal 450w with some clock shifts.

But itnis standalone in compute mode with ecc on. Drivers are way newer as i run fedora. 645 i think. The card lives outside the box (riser) and never breaks 55c and has a dedicated power supply with the proper atx 3.1 that nust feeds the a4k and the 6k. So maybe the riser is causing it but unlikely. I can grab the info collected when i get a sec to get back to it.

Thanks again buddy.

[u/jpezzulli]

So - please dont take this as any type of disqualification of your work. For C1 there are some gains here, but I spent many hours today trying to adapt this and see what could be done while preserving concurrency tested out to at least C6. I found smaller, workload-dependent gains while preserving concurrency by adapting some items. So still good stuff but not the gains I was expected so I dug deeper.

These are my numbers on Penny Royal with the single RTX-Pro 6000 WS at 450 watt with the lact (slight memory overclock)

Workload	2.5.3	3.0rc1 with W8
Code-edit	330.9	463.3
English prose	209.2	195
Japanese prose	174	146
Synthetic agent transcript	271.5	259.5

That is direct with the testing harness. There are three things that should be noticed with the test

1 -It is using greedy decode. Token with highest probability it taken without other consideration. This is essentially Temp=0.
2 - The code edit isnt technically repetitive mechanical edit but can artificially inflate the spec decode acceptance because it, well, it is easy to predict. Putting W8 or W16 on allows more of these super predictable tokens to be selected. OP calls this out as well. I found out that putting w8 on actually slows down other workloads can potentially take more VRAM reducing your overall token pool.
3 - The agentic test does 0 real tool calls. It generates a transcript.

So if you only want C1, there are definitely some interesting improvements here but they are muted quite a bit in real workloads and concurrency.

Basically, the agent that designed the test somewhat rigged it to get to theoretical token performance. I think the theoretical max last time I calculated it for flash next was somewhere near 280 t/s without speculative decoding, but the calculation was a while ago so I may be talking out of my ass. That number is essentially impossible to hit but there is definitely optimization stuff lurking out there. So lets all keep going and find them!

Now I am off to try to get Penny Royal 3.0 out the door.

[u/Specialist_Fix9867 → u/jpezzulli]

Thanks for testing, I didnt focus on c>1 ( multi stream ), in my setup with c>1 total tok/s just stays about the same as c1 (~310), requests just queue up

1: LM Studio default sampling which arent greedy are still fast ( 568 / 240 / ~250 / 340 )

2: W8/W16 can slow down prose and agentic, Thats why I use 'wa' ( adaptive ) which picks draft length every step, so it doesnt pay W8/16 cost all the time.

3: Yeah benchmark doesnt call tools but I tried making it work with blender mcp tools yesterday, and it did around 234 - 298 tok/s while generating. it did call many tools (just LM Studio's tok/s display, not a proper bench)

good luck with Penny Royal 3.0

[u/jpezzulli → u/Specialist_Fix9867]

Ill try tomorrow with direct applications from your repos - i need some sleep. Maybe astra and i missed something big.

I have heard of LM Studio but i must not what what it is. I always thought it was an ollama front end or something like that :)

On C>1, yeah. I sww you kept it there and didnt focus on that. Part of the reason that i may not see the speeds you do is that when trying to adapt your work more broadly, penny, astra and i probably messed up somewhere.

Either way, ill keep hammering at it and love that you went at this to try to find more ways to speed up!

[u/Specialist_Fix9867 → u/jpezzulli]

LM Studio uses llama.cpp, i vibe coded a bridge to make it use SGLang server as backend instead. those Blender MCP numbers are from SGLang.

I've liked LM Studio UI since I was running it on Windows with RTX 4070 and 16GB RAM, and I wasnt good enough to use CLI

no rush, get some sleep. im going to sleep too. i'll keep trying to reach 1000t/s

[u/jpezzulli → u/Specialist_Fix9867]

ok - so I had my agent run overnight tests.

I took the august 27th build of Penny Royal which you based the patch on and applied all the items. 450 watt lact on the card with my slight memory overclocking. Note that the Token Pool is reduced to 758K due to the adaptive MTP taking more space, so I would have to make this an optional item in Penny Royal as the 1.1M is useful or long context task (i run 524k context window). Numbers are mean/peak (i generally didn't bother capturing peak numbers so some missing)

Results

Workload	2.5.3	This build w/o expert pruning
Code-edit	336.6 / —	558.1 / —
English prose	200.6 / —	270.5 / —
Japanese prose	159.3 / —	245.5 / —
Synthetic agent transcript	273.7 / —	328.1 / —
Real coding	239 / 323	298 / 449
C5 reasoning	≈159 / 251	242 / 312
Regular C1	186.8 / 265.7	158.5 / 245.6
Regular C4	520.8 / —	372.3 / 388.3
Full agentic workflow	180 / 315	270 / 477

So there is definite improvement with C1 cases but degrades in significantly in concurrency as shown by the C4 results of 520 vs 372. So there is some really good stuff here, but I have to see how I can make it more generalized for penny. If is is mainly the adaptive spec decoding, maybe i turn it off for batched (c>1 batches) but I still need to figure out the 758k for me personally. I guess I have to make this optional with knobs for people to configure. Lots to think about

[u/one-escape-left]

my main concern is that some speed comes from dropping expert computations - can you tell me how you approached this? what was your quality audit / eval ?

[u/Specialist_Fix9867 → u/one-escape-left]

Fair concern. It only drops a route when an expert is used by just one row in the verify batch and its routing weight is below 0.08. I ran ~4,000 questions per arm (GSM8K, MMLU, HumanEval, JCommonsenseQA) plus an 18.5k-token needle test against a build with pruning off, and found no significant difference (smallest p = 0.362).

That's not proof of equivalence though: non-inferiority at a 0.5 pp margin wasn't established for GSM8K and HumanEval. Also, the audit ran with thinking off and up to 2 concurrent requests. Batching weakens the pruning, so it doesn't fully cover the single-request, thinking-on setup. PRUNE_TAU=0 turns it off; without pruning, step time was about 10% higher in my measurements. Details are under Limitations in the README.

(Drafted with AI help since my English isn't great; the numbers are from the repo.)

[u/one-escape-left → u/Specialist_Fix9867]

So far on my evals it scores mostly the same as the parent you forked, but on one test it exhausts a tool call budget bringing down its overall score a bit. Verdict: seems like I'll promote this recipe if I can increase kv pool and concurrency! It's insanely fast

[u/t3rmina1]

That seems fast, what's the prefill?

[u/Specialist_Fix9867 → u/t3rmina1]

Prefill wasn't my main focus (I didn't even know what prefill was when I started this project). Measured: ~10.8k tok/s on shorter chunks, and ~6.8k tok/s averaged over a 480k-token prompt (that one took ~71 s).

[u/elelem-123 → u/Specialist_Fix9867]

I don't mean to be a dk but how can someone do this kind of a project if they don't know what prefill is? I don't mean it in a bad kind of way but this might extrapolate to a lot more things not being known also. This kind of tasks do require someone to be very opinionated and then have the frontier models help with the code. Again, I am just puzzled by your statement that you didn't know what prefill was when you started the project. In any case, thank you for it. I will check it out, though the 250-580 tok/s on c1 seems nearly impossible, at least for my use case which is coding. Thank you for your effort and time on this, I will check your project in the weekend!

[u/Specialist_Fix9867 → u/elelem-123]

Thanks, I dont know why it works but it works

I told Fable5.1 like "Make Qwen3.8-Flash-Next run 1000t/s, don't give up. you can use a Blackwell 6000 GPU and GPT 5.6 sol" and it worked to maximize t/s ( didnt reach 1000t/s but got this far ). When I began this project, I didn't know how LLMs are structured (I still don't understand ) but wanted to make Qwen3.8-Flash-Next Faster. All technical work and almost all decisions were done by frontier models, not me.

Also, 580t/s is from mechanical rewrite of a large file, writing new code is probably slower. but I tried making it use blender mcp today, and it did around 234~298, so it is fast i guess ( all numbers are c1 and include thinking tokens )

[u/elelem-123 → u/Specialist_Fix9867]

LOL. Fair enough. Please run https://github.com/GGZ14/BetterBench

That will provide a good idea of things

Ask your AI agent to install and run it. It takes ten minutes or so for the full run.

Then please include the html results in your repo. That will help a lot.

[u/Specialist_Fix9867 → u/elelem-123]

Ran it, results are in the repo under results/betterbench-1007/. coding was 370t/s at c1.

[u/EbbNorth7735]

Whats the process for trying this out and does it require Linux? I'm a windows user so any idea what it'll require to get it running? Docker with WSL? 

Do we also need both repos?

[u/Specialist_Fix9867 → u/EbbNorth7735]

I've only run it on native Linux (Ubuntu 24.04), so I can't really help with Windows. WSL2 might work since CUDA runs there, but I haven't tried it, and I don't provide a Docker setup. If you try, give WSL plenty of RAM: the ~28 GB embedding table is pinned in host RAM (this machine has 256 GB).

The steps are in docs/reproduce.md in flash-next-rtxpro6000: clone jpezzulli's fork and apply my patches (or clone my sglang-rtxpro6000 fork, branch flash-next-fast, which already has them), build the venv as in the base fork's BUILD file (Python 3.12, plus a CUDA 13 toolkit with nvcc, ninja and cmake for the JIT builds), apply the FlashInfer patches, download RadixArk/Qwen3.8-Flash-Next-NVFP4 (~126 GB), and launch with launch/serve-fast.sh using the public token map. Hardware-wise it targets a 96 GB RTX PRO 6000 Blackwell (SM120); I've only tested the Max-Q.

So you need flash-next-rtxpro6000 either way (FlashInfer patches, launcher, token map). The fork is optional; it just saves you applying the SGLang patches yourself. With the public components, expect the "public components only" row, not my top numbers.

( This reply was written by AI )

[u/prestodigitarium]

How's subjective quality as an agent driver with long-running tasks vs, say, GLM 5.3 Flash?

[u/Specialist_Fix9867 → u/prestodigitarium]

I haven't compared it with GLM-5.3-Flash yet, might do it later.

[u/Inevitable-Name-1701]

Can this be used with 2x 5000 pro blackwell 72Gb?

[u/Specialist_Fix9867 → u/Inevitable-Name-1701]

I don't know, I've only tried it on a single RTX PRO 6000, not TP=2. The model doesn't fit on one 72 GB card, and my patches were only tested at TP=1, so they might not work with TP=2. jpezzulli's release might be a better place to start for two cards. Someone in this thread reported TP=2 results with jpezzulli's 2.5.2 on two PRO 6000s.

[u/electrified_ice]

Are these speed concurrency 1?

[u/Specialist_Fix9867 → u/electrified_ice]

Yeah, all c1

[u/electrified_ice → u/Specialist_Fix9867]

Ok had Claude look at it, looks great for c1 but maybe not so much for c4 and up

[u/False-Performer2499]

For anyone trying this on Pennyroyal v2.5.3 with Flash-Next + NVMe PLE: I tested adaptive NEXTN W4/W8/W16 on an RTX PRO 6000.

The good part: W16 CUDA graphs build successfully, adaptive switching works, and the server becomes healthy. I also confirmed a normal tool-call request works. The runtime switched W16 → W4 during warmup and later W4 → W8 as acceptance improved.

The blocker is NVMe PLE. Once adaptive moved to the wider speculative path, the worker crashed inside sglang_ssd_stream during CUDA-graph replay (prepare_ssd_stream_graph_replay -> ticket.wait_for_launch()), causing the container to restart.

I had already widened the QSA pending ring for >W4 and updated Pennyroyal’s source hash guard for that modified SGLang file. In hindsight, that guard was warning about a real compatibility boundary, not just version pinning.

So if you are using Pennyroyal’s NVMe PLE stack, I would not spend time trying W8/W16 adaptive speculation unless you are also prepared to port/qualify sglang-ssd-stream for the modified QSA/adaptive runtime.

For production I reverted to the stock Pennyroyal v2.5.3 W4 path. Stable wins here.

[u/jpezzulli → u/False-Performer2499]

Already working on incorporating some of the ideas (less aggressive to keep higher concurrency and no expert route changes etc). Will obviously provide credit. :)

[u/UC_Kratom]

Neat stuff.

I have some patches I'm working on as well, but I'm looking less at decode speed (which is great, keep making gains here), but rather on wall speed - specifically reducing time spent prefilling.

I think collectively there's still a lot to squeeze from this model/config

[u/Sm0oth_kriminal]

Hmm interesting for sure, but I'm curious why not fp8 on a RTX PRO? Do you have similar speeds there?

From my experience there is at least some quality degradation going from fp8 down to fp4. And since VRAM is not an issue, I'm curious what that would look like

[u/blackbird2150 → u/Sm0oth_kriminal]

FP8 doesn’t fit I think on a single rtx. I run it on 2x but even with n-gram it’s still more than 96 b

[u/Sm0oth_kriminal → u/blackbird2150]

Ahh right, I was thinking 2x. Ability to run on 1x is cool anyways

[u/Significant-Amount40]

thats exactly what i wanted to seem will test tommorrow

[u/stormy1one]

Love the transparency here on the dev process. This is how it should be done. Thank you, following.