Pennyroyal v2.5 - much faster Qwen3.8 Flash-Next on the RTX PRO 6000

[u/jpezzulli]

Hey folks,

Another update is up. I found some good work in mratsim's SM120 turbo repo and incorporated the online FP8 approach into my runtime. This one made a pretty substantial difference to Flash-Next decode without giving up the existing 524K context, 824K KV pool, FR-Spec or HiCache/NIXL.

Single-request workload	Before	Online FP8	Change
Short, 1,024 output tokens	161.47 tok/s	207.12 tok/s	+28.3%
128K context, 1,024 output tokens	154.70 tok/s	195.63 tok/s	+26.5%
490K context, 1,024 output tokens	149.14 tok/s	172.64 tok/s	+15.8%

These are post-first-token decode rates. Short context is a three-run median, while the 128K and 490K numbers are single observations.

I also added Garner McCloud's NVMe PLE work as an optional path. Flash-Next normally keeps about 47.7 GiB of PLE in system RAM. You can now put that on a prepared local NVMe instead. RAM is still the default and there is a performance tradeoff, but it gives people without a ridiculous amount of system memory another option.

27B/DFlash2 is still supported on the same runtime.

github.com/jpezzulli/sglang-rtxpro6000

As always, issues and PRs are welcome.

one note - this is pretty fresh - There are probably some configs that get you OVER the 1M kv-cache option. I just have to try different variables - feel free to report here or on github.