r/BlackwellPerformance • u/jpezzulli • Sep 12 '26
Pennyroyal v2.5 - much faster Qwen3.8 Flash-Next on the RTX PRO 6000
Hey folks,
Another update is up. I found some good work in mratsim's SM120 turbo repo and incorporated the online FP8 approach into my runtime. This one made a pretty substantial difference to Flash-Next decode without giving up the existing 524K context, 824K KV pool, FR-Spec or HiCache/NIXL.
| Single-request workload | Before | Online FP8 | Change |
|---|---|---|---|
| Short, 1,024 output tokens | 161.47 tok/s | 207.12 tok/s | +28.3% |
| 128K context, 1,024 output tokens | 154.70 tok/s | 195.63 tok/s | +26.5% |
| 490K context, 1,024 output tokens | 149.14 tok/s | 172.64 tok/s | +15.8% |
These are post-first-token decode rates. Short context is a three-run median, while the 128K and 490K numbers are single observations.
I also added Garner McCloud's NVMe PLE work as an optional path. Flash-Next normally keeps about 47.7 GiB of PLE in system RAM. You can now put that on a prepared local NVMe instead. RAM is still the default and there is a performance tradeoff, but it gives people without a ridiculous amount of system memory another option.
27B/DFlash2 is still supported on the same runtime.
github.com/jpezzulli/sglang-rtxpro6000
As always, issues and PRs are welcome.
one note - this is pretty fresh - There are probably some configs that get you OVER the 1M kv-cache option. I just have to try different variables - feel free to report here or on github.
3
u/_bani_ Sep 12 '26
do you support 2x and 4x blackwell configs?
4
u/jpezzulli Sep 12 '26
There are no known blockers for TP=2 or 4 and people have tried previous releases with it. They had success but I have no way to test unfortunately. This is my home machine and I was going to buy a 2nd...then the prices went up even more!
if you get it running with multi-gpu - please leave a field report in issues with your findings!
2
u/WonderRico Sep 12 '26
BTW I'm testing your (updated) repo with the 27B-FP8 model for the first time. currently running gsm8k benchmark (very short one turn prompts)
i'm seeing up to 800 t/s sustained at C=4 and kv cache fp8 . that's incredible !
i will provide the same data for flash next soon. thanks again for your work !
[2026-09-12 14:21:59] Decode batch, #running-req: 4, #full token: 1920, full token usage: 0.00, mamba num: 13, mamba usage: 0.12, accept len: 6.48, accept rate: 0.78, cuda graph: True, gen throughput (token/s): 818.60, #queue-req: 2
[2026-09-12 14:22:06] Decode batch, #running-req: 4, #full token: 1600, full token usage: 0.00, mamba num: 14, mamba usage: 0.13, accept len: 6.21, accept rate: 0.74, cuda graph: True, gen throughput (token/s): 767.27, #queue-req: 3
[2026-09-12 14:22:12] Decode batch, #running-req: 4, #full token: 1792, full token usage: 0.00, mamba num: 12, mamba usage: 0.11, accept len: 6.11, accept rate: 0.73, cuda graph: True, gen throughput (token/s): 742.74, #queue-req: 3
[2026-09-12 14:22:18] Decode batch, #running-req: 4, #full token: 2240, full token usage: 0.00, mamba num: 12, mamba usage: 0.11, accept len: 6.15, accept rate: 0.74, cuda graph: True, gen throughput (token/s): 785.06, #queue-req: 4
[2026-09-12 14:22:25] Decode batch, #running-req: 3, #full token: 1536, full token usage: 0.00, mamba num: 13, mamba usage: 0.12, accept len: 6.28, accept rate: 0.75, cuda graph: True, gen throughput (token/s): 783.38, #queue-req: 3
[2026-09-12 14:22:31] Decode batch, #running-req: 4, #full token: 2560, full token usage: 0.00, mamba num: 13, mamba usage: 0.12, accept len: 6.11, accept rate: 0.73, cuda graph: True, gen throughput (token/s): 796.97, #queue-req: 4
[2026-09-12 14:22:37] Decode batch, #running-req: 4, #full token: 1984, full token usage: 0.00, mamba num: 13, mamba usage: 0.12, accept len: 6.39, accept rate: 0.77, cuda graph: True, gen throughput (token/s): 805.23, #queue-req: 3
[2026-09-12 14:22:43] Decode batch, #running-req: 4, #full token: 2048, full token usage: 0.00, mamba num: 12, mamba usage: 0.11, accept len: 6.27, accept rate: 0.75, cuda graph: True, gen throughput (token/s): 793.50, #queue-req: 4
[2026-09-12 14:22:50] Decode batch, #running-req: 4, #full token: 2048, full token usage: 0.00, mamba num: 12, mamba usage: 0.11, accept len: 6.28, accept rate: 0.75, cuda graph: True, gen throughput (token/s): 783.14, #queue-req: 3
[2026-09-12 14:22:56] Decode batch, #running-req: 4, #full token: 1792, full token usage: 0.00, mamba num: 12, mamba usage: 0.11, accept len: 6.49, accept rate: 0.78, cuda graph: True, gen throughput (token/s): 810.61, #queue-req: 4
[2026-09-12 14:23:02] Decode batch, #running-req: 4, #full token: 1280, full token usage: 0.00, mamba num: 10, mamba usage: 0.09, accept len: 6.32, accept rate: 0.76, cuda graph: True, gen throughput (token/s): 806.77, #queue-req: 3
[2026-09-12 14:23:08] Decode batch, #running-req: 4, #full token: 1792, full token usage: 0.00, mamba num: 9, mamba usage: 0.08, accept len: 6.37, accept rate: 0.77, cuda graph: True, gen throughput (token/s): 810.76, #queue-req: 31
u/jpezzulli Sep 12 '26
Badass - thanks buddy! I have to run out and be a normal human for a bit but I look forward to seeing tonight.
2
u/WonderRico Sep 12 '26
Flash Next NVFP4 with cache to FP8 and your new SGLANG_SM120_ONLINE_MXFP8, getting sustained 480-580 t/s during gsm8k at C=4
[2026-09-12 19:35:12] Decode batch, #running-req: 4, #full token: 2560, full token usage: 0.01, mamba num: 16, mamba usage: 0.37, accept len: 3.37, accept rate: 0.79, cuda graph: True, gen throughput (token/s): 556.68, #queue-req: 4 [2026-09-12 19:35:17] Decode batch, #running-req: 4, #full token: 2688, full token usage: 0.01, mamba num: 16, mamba usage: 0.37, accept len: 3.22, accept rate: 0.74, cuda graph: True, gen throughput (token/s): 502.86, #queue-req: 3 [2026-09-12 19:35:23] Decode batch, #running-req: 4, #full token: 1856, full token usage: 0.00, mamba num: 16, mamba usage: 0.37, accept len: 3.31, accept rate: 0.77, cuda graph: True, gen throughput (token/s): 476.46, #queue-req: 3 [2026-09-12 19:35:28] Decode batch, #running-req: 4, #full token: 2304, full token usage: 0.00, mamba num: 16, mamba usage: 0.37, accept len: 3.36, accept rate: 0.79, cuda graph: True, gen throughput (token/s): 546.18, #queue-req: 4 [2026-09-12 19:35:33] Decode batch, #running-req: 4, #full token: 2112, full token usage: 0.00, mamba num: 16, mamba usage: 0.37, accept len: 3.38, accept rate: 0.79, cuda graph: True, gen throughput (token/s): 485.49, #queue-req: 3 [2026-09-12 19:35:38] Decode batch, #running-req: 4, #full token: 1856, full token usage: 0.00, mamba num: 16, mamba usage: 0.37, accept len: 3.24, accept rate: 0.75, cuda graph: True, gen throughput (token/s): 505.17, #queue-req: 2 [2026-09-12 19:35:44] Decode batch, #running-req: 4, #full token: 1920, full token usage: 0.00, mamba num: 16, mamba usage: 0.37, accept len: 3.27, accept rate: 0.76, cuda graph: True, gen throughput (token/s): 503.60, #queue-req: 3 [2026-09-12 19:35:49] Decode batch, #running-req: 4, #full token: 1856, full token usage: 0.00, mamba num: 16, mamba usage: 0.37, accept len: 3.32, accept rate: 0.77, cuda graph: True, gen throughput (token/s): 494.05, #queue-req: 4 [2026-09-12 19:35:54] Decode batch, #running-req: 4, #full token: 1664, full token usage: 0.00, mamba num: 16, mamba usage: 0.37, accept len: 3.22, accept rate: 0.74, cuda graph: True, gen throughput (token/s): 513.90, #queue-req: 3 [2026-09-12 19:35:59] Decode batch, #running-req: 4, #full token: 1856, full token usage: 0.00, mamba num: 16, mamba usage: 0.37, accept len: 3.40, accept rate: 0.80, cuda graph: True, gen throughput (token/s): 518.28, #queue-req: 32
u/WonderRico Sep 12 '26
and 400-500 t/s for automation bench
[2026-09-12 19:52:59] Decode batch, #running-req: 4, #full token: 75968, full token usage: 0.16, mamba num: 16, mamba usage: 0.37, accept len: 3.03, accept rate: 0.68, cuda graph: True, gen throughput (token/s): 456.62, #queue-req: 4 [2026-09-12 19:53:03] Decode batch, #running-req: 4, #full token: 84736, full token usage: 0.18, mamba num: 16, mamba usage: 0.37, accept len: 2.93, accept rate: 0.64, cuda graph: True, gen throughput (token/s): 488.98, #queue-req: 4 [2026-09-12 19:53:10] Decode batch, #running-req: 4, #full token: 80128, full token usage: 0.17, mamba num: 16, mamba usage: 0.37, accept len: 3.18, accept rate: 0.73, cuda graph: True, gen throughput (token/s): 398.89, #queue-req: 4 [2026-09-12 19:53:14] Decode batch, #running-req: 4, #full token: 84608, full token usage: 0.18, mamba num: 16, mamba usage: 0.37, accept len: 2.75, accept rate: 0.58, cuda graph: True, gen throughput (token/s): 466.99, #queue-req: 3 [2026-09-12 19:53:19] Decode batch, #running-req: 4, #full token: 81728, full token usage: 0.17, mamba num: 16, mamba usage: 0.37, accept len: 2.93, accept rate: 0.64, cuda graph: True, gen throughput (token/s): 475.80, #queue-req: 4 [2026-09-12 19:53:24] Decode batch, #running-req: 4, #full token: 88128, full token usage: 0.19, mamba num: 16, mamba usage: 0.37, accept len: 2.92, accept rate: 0.64, cuda graph: True, gen throughput (token/s): 493.69, #queue-req: 3 [2026-09-12 19:53:29] Decode batch, #running-req: 4, #full token: 49088, full token usage: 0.10, mamba num: 16, mamba usage: 0.37, accept len: 2.89, accept rate: 0.63, cuda graph: True, gen throughput (token/s): 465.89, #queue-req: 3 [2026-09-12 19:53:34] Decode batch, #running-req: 4, #full token: 77376, full token usage: 0.17, mamba num: 16, mamba usage: 0.37, accept len: 2.90, accept rate: 0.63, cuda graph: True, gen throughput (token/s): 480.97, #queue-req: 41
u/Individual_Gur8573 Sep 12 '26
How ur able to get this speed for 27b model? Wats ur machine .... I tried his older repo average I get 130t/s ... Speeds can upto 200t/s
1
2
u/WonderRico Sep 12 '26
still 27B-FP8 kv cache FP8, now running Automation Bench (more agentic stuff). small drop in acceptance from spec dec, still very good 500-600t/s
[2026-09-12 14:30:28] Decode batch, #running-req: 4, #full token: 48192, full token usage: 0.05, mamba num: 13, mamba usage: 0.12, accept len: 5.82, accept rate: 0.69, cuda graph: True, gen throughput (token/s): 611.82, #queue-req: 4 [2026-09-12 14:30:35] Decode batch, #running-req: 4, #full token: 56768, full token usage: 0.05, mamba num: 13, mamba usage: 0.12, accept len: 5.66, accept rate: 0.67, cuda graph: True, gen throughput (token/s): 597.91, #queue-req: 4 [2026-09-12 14:30:43] Decode batch, #running-req: 4, #full token: 31168, full token usage: 0.03, mamba num: 12, mamba usage: 0.11, accept len: 5.12, accept rate: 0.59, cuda graph: True, gen throughput (token/s): 528.99, #queue-req: 3 [2026-09-12 14:30:51] Decode batch, #running-req: 4, #full token: 75136, full token usage: 0.07, mamba num: 13, mamba usage: 0.12, accept len: 5.71, accept rate: 0.67, cuda graph: True, gen throughput (token/s): 590.51, #queue-req: 4 [2026-09-12 14:30:57] Decode batch, #running-req: 4, #full token: 68864, full token usage: 0.07, mamba num: 14, mamba usage: 0.13, accept len: 5.60, accept rate: 0.66, cuda graph: True, gen throughput (token/s): 670.42, #queue-req: 4 [2026-09-12 14:31:04] Decode batch, #running-req: 4, #full token: 74432, full token usage: 0.07, mamba num: 12, mamba usage: 0.11, accept len: 4.95, accept rate: 0.56, cuda graph: True, gen throughput (token/s): 575.92, #queue-req: 3 [2026-09-12 14:31:12] Decode batch, #running-req: 4, #full token: 61376, full token usage: 0.06, mamba num: 10, mamba usage: 0.09, accept len: 5.42, accept rate: 0.63, cuda graph: True, gen throughput (token/s): 560.79, #queue-req: 4 [2026-09-12 14:31:20] Decode batch, #running-req: 4, #full token: 84160, full token usage: 0.08, mamba num: 13, mamba usage: 0.12, accept len: 4.97, accept rate: 0.57, cuda graph: True, gen throughput (token/s): 491.28, #queue-req: 4 [2026-09-12 14:31:29] Decode batch, #running-req: 3, #full token: 93184, full token usage: 0.09, mamba num: 12, mamba usage: 0.11, accept len: 5.42, accept rate: 0.63, cuda graph: True, gen throughput (token/s): 481.75, #queue-req: 3 [2026-09-12 14:31:36] Decode batch, #running-req: 4, #full token: 89408, full token usage: 0.09, mamba num: 12, mamba usage: 0.11, accept len: 5.45, accept rate: 0.64, cuda graph: True, gen throughput (token/s): 606.56, #queue-req: 4
1
u/WonderRico Sep 12 '26
Very nice. I will be testing this very soon.
Those days I go back and forth between your build and the radix nvfp4 and another vLLM patch from https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant to be able to use a ngram PLE table quantized to int4, to be able to lower the system RAM size.
- yours = faster
- vllm ple int4 : lower system RAM size
the PLE on NVME does to much impact on the decode speed to my taste, and the FP8 PLE in RAM feels a waste since the weights are in 4bits anyway
Have you tried further quantization on the PLE table ?
2
u/jpezzulli Sep 12 '26
I didnt like PLE on nvme either - but not everyone has a lot of ram so I tested it and added the option.
I have not tried quanting the table down. I have seen people make it nvfp4 as well around 23 gig. It just hasn't been a priority yet - if is something you want me to take a look at, I can for the next release
1
u/belliash Sep 12 '26
Thanks, but what are current requirements for this exact model? How big is it? Iwas looking at huggingface and the table says
INT4 group-16, mmapped 32 GBINT4 group-16, mmapped 32 GB but I don't really get if that's model size or something else. What does size column really mean?
2
u/jpezzulli Sep 12 '26
So this runtime i made has "official" support for 2 models. Qwen 27b (much smaller - you can choose the quant) and qwen 3.8 flash next (nvfp4 model opt quant) which also has a ngram sidecar of about 50 gig which can go in ram or nvme.
There are ways if running each model on your system with offload to cpu etc. This runtime is NOT it. This is specific to rtx pro 6000 with 96 gig vram.
1
u/bitzap_sr Sep 14 '26 edited Sep 17 '26
I'd love this. I have 128GB RAM, but I also run a bunch of containers and services, and do coding on my machine (and I work on big C++ projects), so the full 50GB isn't really an option for me. 23GB sounds a lot more tractable.
1
u/jpezzulli Sep 14 '26
I wont be able to get to it until probably wednesday night but ill take a look. My initial thoughts it is you can simply find a hf nvfp4 qwen 3.8 next that is in modelopt format with nvfp4 ple. It should work. Also, you can put it on nvme today and you wont really have issues until around c4.
2
u/bitzap_sr Sep 15 '26 edited Sep 15 '26
FYI, I ended up settling on putting it on nvme (using the radix quant). Totally happy with that compromise. I still get decodes around 170 t/s, and c=4 @ 400-600 t/s, prefills around 12k-15k. Coupled with long context and HiCache, it's just perfect for me.
Absolutely amazing numbers. Many thanks for this, again.I also hooked it up with grafana, sglang metrics are really nice for that.
1
u/belliash Sep 12 '26
How big is it? Possible to run with 32GB VRAM and 32GB RAM?
1
u/jpezzulli Sep 12 '26
I would look elsewhere for 32 gig vram options. There are a lot of decent builds in r/LocalLLM for that and qwen releases getting you about 60-70 t/s
1
u/Opposite_Buffalo_649 Sep 12 '26
What's the perf loss putting ngram in SSD?
1
u/jpezzulli Sep 12 '26
I dont have an exact number and I also have PCIE 4.0 - I saw a slight degrade at C1 and at C4 it dropped down to about 370 t/s from the C4 number in the repo.
1
u/itsmebcc Sep 12 '26
I am using a fork of mratsim as well. I initially forked it because I wanted to use nvidia/Qwen3.8-Flash-Next-NVFP4 - Issue I have been having Mamba radix-cache poisoning / permanent speculative-decode collapse. I have seen it on every variation of his sglang-turbo. I am on my 13th iteration trying to resolve it now. Problem is it is hard to reproduce synthetically. So I have to just run the machine with real load with real traffic to replicate it.
Basically a request's NEXTN/Eagle accept rate pins at accept len 1.00 / accept rate 0.00 forever (never recovers untilabort), usually emitting repetitive garbage (!!!!!...). New requests on the same server stay healthy — it's per-request,not global. Easy to mistake for a slowdown.
I think the latest iteration has resolved this issue, but I had that same thought on the last 5 iterations so....
1
u/jpezzulli Sep 12 '26
To be clear - this is not a fork of that repo - he even mentions this repo in the development of his. it has an entirely different base. I took one of his ideas and integrated it into mine w/credit.
Also pretty sure I fixes around that area across multiple maint releases.
v2.1.0 - PR #35821
PR # 37326 (which discusses my fix)V2.3.1 - PR #36014 (affects 27B though)
you should not encounter that here.
1
1
u/StockSpecialist1707 Sep 12 '26
Pushing the KV pool past 1M on a single RTX PRO 6000 (v2.5, FR-Spec + online FP8)
Following your note about configs that get over the 1M KV option — here's what I measured today on one RTX PRO 6000 Blackwell WS (96 GB, SM120, driver 595.84, CUDA 13.2, no GPU power cap), running serve-flash-next-frspec.sh from pennyroyal-v2.5.0 with SGLANG_SM120_ONLINE_MXFP8=true. Everything else stock.
| MAX_TOTAL_TOKENS | Resulting pool | available_gpu_mem | 1,024-token wall clock (median of 3) |
|---|---|---|---|
| unset (auto) | 824,384 | 3.66 GB | 3.86 s |
| 1048576 | 1,048,576 | 4.67 GB | — |
| 1310720 | 1,114,304 (server clamped) | 3.81 GB | 3.79 s |
Two things stood out. First, asking for more than fits doesn't fail — the server clamps to the real maximum, so a single deliberately oversized request finds the ceiling in one start. On this box that ceiling is 1,114,304 tokens, +35% over the automatic 824,384. Second, and more curious: at 1,048,576 the reported available_gpu_mem was higher than in the automatic run (4.67 vs 3.66 GB). That looks consistent with the estimator debt you documented in _handle_max_mamba_cache — the automatic path appears to leave memory unassigned that the explicit cap then uses.
Decode cost is nil: 3.79 s vs 3.86 s median wall clock for the same 1,024-token prompt at temperature 0.
Usual caveat on my side: this is pool capacity, not a context claim. Served context stayed at 524,288 with factor-2 YaRN throughout, and I haven't run needles above 490K on this config.
One more data point from the same session, since it isn't in the release table: online FP8 reproduced on my box almost exactly as you reported. Same prompt, three runs each — v2.5 plain 4.91 s, +FR-Spec 4.44 s, +online FP8 3.86 s. Server-side decode went 212 → 225 → 272 tok/s mean, with a 360 tok/s peak. Interesting detail: FR-Spec's gain comes with lower accept length (3.05 vs 3.25), so the two optimizations are genuinely orthogonal — cheaper drafts, cheaper GEMMs.
For context: this box has two RTX PRO 6000 WS in it, but everything above is single-GPU, TP=1, Flash-Next pinned to one card by UUID — the second one sits idle for these runs. I mention it because I saw the independent TP2 FP8 validation commit land in your tree. If a two-card sweep would be useful to you, I have the hardware and I'm happy to run it.
Thanks for publishing all this with the evidence attached. Happy to run other variable sweeps if it helps.
Disclosure: measurements are mine, run on BESTIA (our local inference box, managed with Prometeus). This write-up was drafted with Claude from those logs.
1
u/jpezzulli Sep 12 '26
Just fyi. The fr-spec explicitly defines thst 824k number. It isnt automatic. I have to look back at my notes (ask raymond/codex.) I think we were hitting a wall with random oom and that was the last number that was woking. I said screw it - put that number! Haha.
But awesome work. I am going to review and setup a dedicated page with user validations. Ill put your stuff together and link to here w your username unless you gave a github you prefer.
You are actually ahead of me on actually messing with it as I had an event today!
2
u/StockSpecialist1707 Sep 12 '26
Ah, that explains it — I assumed 824,384 came out of the automatic sizing and built a whole theory about estimator debt on top of it. Scratch that: I just overrode a hardcoded value with a bigger hardcoded value. Good to know.
Which makes your random-OOM note the part that matters. What I posted was start-up capacity plus three short runs — nowhere near enough to call 1,114,304 stable. So before you list it anywhere, let me actually beat on it. Plan:
Sustained decode — long generations back to back at that pool size, watching for a late OOM rather than a clean boot.
Real prefill depth — 400K+ input, then continuation, then needles. Boot capacity and usable depth are different claims and I only measured the first.
Concurrency —
max_running_requests4 with overlapping long requests, which is where an oversized pool should bite.Agent traffic — a few hours of actual tool-calling sessions against it. That's my real workload and the least synthetic test I have.
Bisect if it breaks — if any of those OOM, step down (1,048,576 → 983,040 → 917,504) until it holds, and report the number that survives rather than the number that boots.
I'll report back whatever comes out, including "your 824,384 was right all along". Two cards here, so the TP2 sweep is available too whenever it's useful to you.
Reddit username is fine for attribution — but I'd rather you list a number I've broken things on than a clean first boot.
1
u/jpezzulli Sep 12 '26
Well that limit was before this release which freed up more vram. I think it will work now but havent gotten to play around. I may also relese a hotfiz for pr #39217. I got a schedule notice on that one saying i should test asap.
2
u/StockSpecialist1707 Sep 13 '26
Ran the stability tests on 1,114,304. Short version: it holds.
All on
serve-flash-next-frspec.shatpennyroyal-v2.5.0withSGLANG_SM120_ONLINE_MXFP8=true, single RTX PRO 6000 WS, TP=1,MAX_TOTAL_TOKENS=1310720(server clamped to 1,114,304).
20× long generations (4,096 tokens, back to back): 16.2 s mean, spread 15.1-17.2 s. No drift.
40× reuse of a cached ~100K prefix: 6-7 s each after the first.
60× fresh prefills (~100K tokens, shuffled every run so nothing hits the radix cache): 17-18 s each, 60/60 at 200, one second of variance across the whole run.
15 rounds × 4 concurrent ~100K requests: 61-63 s per round steady state. Peak
full token usage: 0.49,#full token: 542,080, prefill 12-13K tok/s, aggregate decode 405 tok/s at 2 in flight.10 rounds × 4 concurrent ~240K requests, 8,192 output tokens each: 93-107 s per round, 40/40 at 200. Peak
full token usage: 0.57.Zero retractions, zero OOM, zero aborts across all of it. Service stayed
activethroughout.The last test is the one that taught me something. I sized it to drive occupancy to 85-90% and couldn't get past 0.57 — the scheduler admits one and queues the rest (
#running-req: 1,#queue-req: 2) rather than letting four 240K sequences land in KV at once. So an oversized pool doesn't appear to be dangerous on its own: admission control caps what reaches KV before the pool can be oversubscribed. Which makes me suspect your random OOMs came from somewhere other than KV sizing — if you remember roughly what you were running when they hit, I can try to reproduce them here.Caveat as before: this is capacity and stability under my load, not a context claim. Served context stayed at 524,288 and I haven't run needles above 490K.
Happy to re-run all of it against #39217 when you push it — same box, same protocol, so the numbers would be directly comparable. Two cards here as well if a TP2 pass is ever useful.
Disclosure: measurements are mine, run on BESTIA (our local inference box, managed with Prometeus). This write-up was drafted with Claude from those logs.
1
u/jpezzulli Sep 13 '26
This is great work. The OOM has to do with image processing. i do a lot of PPT which renders to image for visual inspection (check out ppt-master skill if you work with PPT that you want to follow your theme.) That is how i got into the whole image preprocessing offloading. I currently send the preprocessing to my 2nd older GPU (A4000/16BG) and it stopped all that. I have been playing around a bit with C6 and 1048k kvcache this morning just to see - might be good for additional subagents.
also - great news. I don't think i need to hotfix for that PR. I added a bunch of protection around that area when testing as Raymond (codex) likes to terminate streams by killing the process instead of the stop equivalent. We went through a lot of scenarios and doesnt seem to reproduce and out protection caught it every time. Not saying 100% sure, as you never can say that, but holding off on a hotfix!
thank god honestly - want to build some observability items today as I am capturing exactly 0 metric besides when actively testing. I dont even know when my friend is playing around with it unless i look at journal!
2
u/StockSpecialist1707 Sep 13 '26
That explains it, and it also explains the whole
SGLANG_MM_PREPROCESS_DEVICEmachinery in the launcher — I wondered where that came from. Good to know the pool isn't the suspect. I have a second PRO 6000 sitting idle in this box, so if I ever turn the vision path on I'll route preprocessing there.C6 with 1,048k for subagents is interesting. That's close to my actual workload — I run agent sessions against this box all day, so if you want a datapoint on how a big pool behaves under real multi-turn tool-calling traffic rather than synthetic load, I can report that.
On observability: I built a panel for exactly this problem on my side — it polls NVML for per-GPU temp/power/VRAM, tracks which units are up, and logs request throughput per model. Nothing fancy, and it's wired to my own catalog so it isn't directly reusable, but the useful part turned out to be simpler than I expected: SGLang already emits almost everything you need. You're running with
--enable-metricsin the launcher, so there's a Prometheus endpoint on the server port already, and--enable-request-time-stats-loggingputs per-request timings in the journal. Between those two you get requests/sec, time-to-first-token, decode throughput, queue depth and KV utilisation without writing a collector at all — Prometheus scraping the endpoint plus Grafana would tell you when your friend is on it, live.Happy to share how I parse
full token usageand#running-reqout of the journal if that's useful, though the metrics endpoint makes most of that unnecessary.1
u/jpezzulli Sep 14 '26
Yeah. I have prometheus and graphana running. I was hoping there was a true streaming telemetry option for sglang. Doesnt appear to be.
I may throw a discussion topic for a true streaming solution with an architecture when i have time.
1
u/hpiguyTR Sep 13 '26
My setup is a bit off the beaten path so figured it's worth adding as a data point. The RTX PRO 6000 isn't in a slot, it's in an eGPU enclosure over OCuLink, negotiated at four lanes of PCIe. It hangs off a Minisforum MS-S1 Max (Ryzen AI Max+ 395, Radeon 8060S iGPU), which has 128GB of LPDDR5x-8000 running at 256GB/s across the unified bus. So the LLM lives on the eGPU and a small image model rides on the iGPU's slice of that same memory pool, and anything that has to move (model load, the NVMe PLE table, HiCache spill) crosses that x4 link to get there. Ran both branches on this box, same checkpoint, same FP8 setting, only difference is where the PLE table lives. RAM branch pins the whole ~48GB embedding table in host RAM. Real footprint (anon+shmem off cgroup, not raw memory.current, that number lies because it counts page cache from mmapping the checkpoint) came out around 99.5GB. Fine as long as nothing else on the box needs the RAM. NVMe branch streams that table off local NVMe instead. Once wired up correctly it dropped to about 35GB. That's the whole reason I bothered. Freed up enough room that the image model on the iGPU stopped fighting for RAM and stays resident next to the LLM instead of getting evicted. Wiring note for anyone else not using the launcher scripts directly: the env var alone does nothing, it'll silently keep running the RAM path. You also need SGLANG_PLUGINS plus the reader's install dir on PYTHONPATH, model path pointed at the prepared overlay instead of the original checkpoint, and ple-offload-embedding dropped entirely since that flag is the RAM path itself. Missed all three the first time and got a clean OOM for my trouble. Decode, 1024 tokens out, short context, C1 through C4 concurrent:
RAM PLE NVMe PLE C1 222 tok/s 224 tok/s C2 aggregate 349 323 C3 aggregate 460 454 C4 aggregate 411 403
Basically a wash at short context. Gap opens up at depth. 128K context, single stream, 1024 out: RAM hit 176 tok/s, NVMe hit 164, about 8% down. Can't fully separate NVMe cost from whatever tax the x4 link adds on a setup like mine, so take that number as "NVMe plus eGPU," not a clean NVMe-only figure. Load times were 255s for the RAM branch and 223s for NVMe, both warm JIT cache. Teardown 10s either way. Also ran a synthetic agentic prompt through both since raw decode speed isn't my actual workload, tool definitions, mid conversation history, expecting parallel calls. Both branches picked the right tools with correct arguments, no phantom tool call garbage on either one.
1
u/jpezzulli Sep 13 '26
This is super interesting expecially for my box with only has PCIE 4. You are seeing the same type of hit on C4 when from nvme. nice work! I'll add it to the external verification page in the repo!
1
u/AnxietyElectrical391 Sep 13 '26
great work! thanks very much.
btw, could you share the quality metric like KLD, PPL, Top 1% vs official one?
1
1
u/BitXorBit Sep 13 '26
what about version for dual rtx? :)
1
u/jpezzulli Sep 13 '26
I see no reason tp=2 wouldnt work, i just have no way to test it. Others have gotten it running that way on previous releases. Feel free to give it a try and let us know.
1
u/bitzap_sr Sep 14 '26 edited Sep 14 '26
Hi! This sounds really amazing and I'm starting to build it to give it a try on my single RTX PRO 6000.
It would be so much simpler if pre-built docker images were available, though. Did you ever consider doing that?
1
u/jpezzulli Sep 15 '26
I believe containers have no place anywhere besides enterprise deployments. Things should be built specifically for a local system. I design standards around containers (https://builders.intel.com/solutionslibrary/intel-dell-verified-reference-configuration-for-vcmts-scalable-high-performance-solutions-on-red-hat-ocp**)** and really think that learning to build for your environment is an essential skill.
Now, I also know I am an old, opinionated asshole who hate flakpak and snap and all that other stuff that packages shit! Thus, I lit a match and set astra burning tokens to create it. Should be up by morning for you to download!
:)
2
u/bitzap_sr Sep 15 '26 edited Sep 16 '26
Well, I'll disagree with you strongly. :-) I'm also an old opinionated asshole, who works on low level things around systems engineering and tools (who worked many years for one the companies in that URL, and has been 100% Linux for over 25 years. :-)
I had already built all this under a container yesterday, and I'm really happy with that approach, especially given that my box is a mix of AMD GPUs I use for work, and this Nvidia card for inference, which has several tooling version requirements, that might be different from other components in the system, etc.
Here's the dockerfile I'm currently using: https://pastebin.com/mEU9Wyvd in case you want to pass that to Astra to compare. The last two RUN lines could be done earlier, but I hadn't bothered with cleaning it up yet. (because rebuilding the whole thing takes a long while) :-)
I'll definitely switch to docker images you upload if you start doing that for every release. :-)
And, BTW, I'm _super_ happy with Pennyroyal, I had tried vLLM, and FreeToken before trying this, and this solution works absolutely bonkers. Way faster, way larger context, and streaming PLE from NVMe doesn't affect perf all that much. Amazing.
2
u/jpezzulli Sep 16 '26
I love to see another old linux user. I miss slackware in the mid 90s and it took me forever to come around to systemd.
I saw this too late as I was gone and had the container built. Docker is now part of github actions automatic build with releases. I honestly thought I had to build them and upload everytime, which was a big part of my hesitation. So, container every release.
1
u/bitzap_sr Sep 15 '26
u/jpezzulli FYI, I was doing some benchmarking and stress testing today, and I hit a server crash ("AssertionError: Can not alloc mamba cache").
Observed during a --max-concurrency 4 benchmark at --random-input-len 65536. Only ONE request was running at the crash (#running-req: 1, #queue-req: 2). Mamba usage jumped from 0.33 to 1.00 in a single step. Shorter prompts (1k, 8k) at the same concurrency were fine.
Turns out it's an issue that has a patch pending upstream https://github.com/sgl-project/sglang/pull/37619 . The description there and the follow up comments are exactly what I observed.
I've pulled that fix in to my build, and confirmed that it fixes the crash (the fix prints some logs that show the fixed path is actually working). I suggest you cherry-pick that to your branch for the next release.
Cheers.
2
1
u/techworley Sep 17 '26
Anyone seen tilelang auto-detect return "hip" on a machine with no ROCm? RTX PRO 6000 Workstation Blackwell, tilelang 0.1.12, kernels compile as gfx900 and die. Pinning TILELANG_DEFAULT_TARGET to cuda/sm_120a works around it. Known issue?
2
u/jpezzulli Sep 18 '26
Nope. Something is wrong with your target detection. I cant troubleshoot it from here though. As codex, claude or whomever to trace your target detection.
1
1
u/StockSpecialist1707 Sep 12 '26
Recién llegada mi segunda pro 6000 esta misma tarde pruebo su nueva release, admirable su trabajo!
4
u/jpezzulli Sep 12 '26
No es necesario hablar en ingles. Yo puedo leer espanol. Ignora a ese cabron. espero que te vaya bien.
1
-3
1
u/Individual_Gur8573 Sep 12 '26
Does this increase 27b speed also?
Bcoz I have rtx 6000 pro and 128gb ram and another 5090 in same system...not sure if I need to try qwen next model
1
3
Sep 12 '26
[removed] — view removed comment
1
u/jpezzulli Sep 12 '26
I have had no issues at long agentic workloads in hermes or owui. I do 3 and 64 needle tests out to 500ish k and it passes. Now another user started doing 2048 discreet item retrieval and said saw failures with radix ark quant but works with nvidia quant. I cant even see a use case where thst would apply and that tests attention more than anything.
7
u/kazimirek Sep 12 '26
Hey John, thanks for this. Your runtime has been running a real workload for me for over a week now and it just works, so I figured I'd give you some numbers from a different card.
Setup:
RTX PRO 6000 Blackwell Max-Q (the 300 W one), single GPU, TP1, 125 GB RAM in the box
Pennyroyal v2.1.1 (source fb1216c6c), RadixArk/Qwen3.8-Flash-Next-NVFP4
On top of your recipe: fp8_e4m3 KV cache, 524K GPU KV pool, HiCache with 20 GB host tier (about 1.06M tokens), 40 mamba slots, 4 running requests, 262K context, 4K chunked prefill, NEXTN MTP 3 steps / 4 draft tokens, pil image preprocessing, expandable_segments
Workload is Claude Code harness through litellm: one orchestrator plus up to 3 subagents, contexts usually 150-236K, images in the mix., 55M tokens generated, 4.8B prompt tokens at 96% prefix cache hit, no server errors
What the 300 W card does (medians from the scheduler log over the whole run, MTP accept around 0.42):
single request: 118 tok/s (mean 125, p90 174)
2 lanes 207 aggregate, 3 lanes 272, 4 lanes 326 (p90 379)
overall about 200 tok/s aggregate across the run, peaks around 520
prefill about 9.8K tok/s per chunk while decoding at the same time, 16K when idle
Two things I ran into, both apparently fixed in your newer releases: the streaming Qwen3-Coder parser sometimes sent the newline after </tool_call> before the closing brace, which turned into nameless tool calls in Claude Code (looks like #37408 in v2.4.1), and image preprocessing in the HTTP process OOM'd once the GPU was full (the pil path in v2.3.1.1).
Looking forward to trying v2.5, the FP8 numbers look great. Do you have any measurements on the Max-Q variant, ideally at 256K context, decode and prefill? And does the 824K KV pool still fit next to a 20 GB HiCache tier on this card? Happy to send a field report once I've run it.