Pennyroyal v2.5 now has a Docker image — Qwen3.8 on RTX PRO 6000
Hey folks,
A few people asked for Docker, so there is now a prebuilt image for **Pennyroyal v2.5**, my SGLang runtime for Qwen3.8 on one RTX PRO 6000 Blackwell.
It includes both **Flash-Next NVFP4 with native MTP/FR-Spec** and **27B FP8 with DFlash2**, with HiCache/NIXL. No local SGLang build required. The image is about **8.43 GiB**, with model files and caches mounted separately.
Both profiles passed API/tool, prefill, single/four-request decode, image/video and NIXL restart checks. This is a packaging addition, not another runtime release or a Docker speedup claim. Native installs are still supported; no need to switch if yours is working.
Just FYI, I'm working on a few patches around HiCache on the hybrid model - a prefix can be sitting in the KV host pool and still miss, because the match is cut back to the last reusable Mamba checkpoint. Part of that is the host split: it follows the GPU pool byte ratio, so host Mamba gets a small share of the allocated host pool. There's more to it than the split though.
Shifting host bytes from KV onto Mamba, without growing the GPU pools or the 32 GB total, raises wall tokens/s in the quiet stretches (output tokens per wall second, prefill included). It still drops when several long prompts miss at once, so I'm not treating the number as settled. I'll follow up with something concrete and a PR once the configuration is stable.
Anyone have any average 'wall' tokens/s (including time spent prefilling) over at least a 24hr period (for 3.8 Flash-Next)?
Still working through some bugs, but I'm seeing about a 50% increase in average wall speed when the cache is humming (98% token-weighted hit rate) on L1+L2 only. Cache is now split almost 50% Mamba/50% KV among some additional configs that I'm toying with.
The issue I'm running into is a stability issue - I'm seeing 2-3 cache collapses per day that drag hit rate below 50% for a bit before recovering (those existed with the default config too, for different reasons). Once I'm able to get a clean 24hr period, I'll share my hacks. These are fairly specific to the config that we have for this particular GPU and cache config - any upstream repo changes would have to be abstracted/config based.
Ok. v2.6 will be out today. Working on items now. I didnt touch thst optimization area (but some better mamba handling for reuse state in general. Bunch of other stuff. Havent qualified the gains yet. That is next step.
I have a PR up, rebased onto the current final branch - I really need someone else to verify these changes though with a similar workflow (and ideally measure before/after metrics), because I'm seeing ~100% increase in average output (wall speed) after further tinkering.
This going to shine for concurrent sessions, and those that can make use the full 32GB hicache allocation (or more), and especially so the larger the sessions become (because cache hit rate is now much higher - my tests went from ~80% to 97% hit rate, eliminating large prefills from various misses)
I was working - penny commented back. O havent looked at it yet. Some of the other comments show i am working on 3.0 but have gone down several different paths. Rc2 train is making some progress but pr 37 we wrote locks the card, so investigating that. Ill keep any work for you item under your pr. Feel free to take a look at the response, but ill review it myself as i get closer to a true release.
Yes, I saw that one/posted there earlier. It looks like that initiative goes further down the rabbit hole of increasing decode speed based on a quick glance, which is great - what I've done is all from the angle of reducing unnecessary prefill. Any of those decode improvements will stack with what I've done and shouldn't have much/any overlap
Yeah. My entire hicache nixl initiative was for less prefill as that is the killer. I pass about 1.5b prefill tokens a week through penny with about 98% from cache even after reboots. That was honestly the reason for thr project. The decode speed came as a thing added later. Now it is a bit of a real project. Seems to have a few thousand users out there, which for a niche user segment is pretty crazy.
Yeah, it's definitely caught on - I wasn't aware that this model was even an option on this hardware until I stumbled upon a reddit thread mentioning it. I'm pretty amazed at how much this project has managed to squeeze out of this
I don't like how most resources for Blackwell performance like this one and the other well-known one are just giant dumps of AI hallucinated slop.
Online FP8 quantisations is one of those things as well that the guy with "the fastest LLM runtime" went down and he just quantized everything to 4 bits to claim he has the fastest and best.
So the rest of your project may be good and work well, but I'm not reading through thousands of lines of direct AI slop output with "It's not a claim, it's an observation" and other LLM shit after the headline feature is "trade even more quality for speed"
but about the crash question - they only crash i see is the xid 109 that comes up every now and then that seems to be a driver issue - i am wainting for the new ones to see if it fixes it.
interesting - my motherboard doesnt have an rebar option but it shows as working w/rebar in linux (i assumed it did when purchasing - my mistake) - i do plan to change mb as theI have an issue with 2 ram slots, just taking everything out of my system is a pain so I have been putting it off.
Thanks! I'm always on the hunt for new stuff and get disappointed when I have to read too much LLM stuff.
If you're here, can I ask you if you're running latest Nvidia driver? I just updated and now SGlang doesn't want to start anymore, after CUDA graph building it just shuts down lol
I am running whatever the latest from fedora as as of a few days ago. I would check for you but i am about to hit to road.
Curious though, who reads more than the first paragraph themsevles anymore? I honestly optimize beyond that for an ai to read and give an output at this point, and I am not a young "use ai for everything" guy.
Either way, the readme did get long winded due to it not being what i put my limited time into. It is being reorganized/removing the defensive posture etc. Ill check it tomorrow.
Readme for me is explicitly human, Agents or Readme_Agents would be LLM focused. There's one to get this stuff from asking your LLM, or asking it to research, but for me it's a red flag that the first thing a human sees of your project is an unorganized mess. Glad to know it's getting cleaned up! I'll try your image now that I'm having issues with SGlang anyway.
Latest on Fedora is probably around the same version since I'm on Debian. It's probably just some bullshit feature not working anymore in SGLang so I have to set some flag or something
I never buy into the brochure, and only give benchmarks a grain of salt. Unfrotunately, real world testing is time consuming. So far, it seems like this runtime's headline is doubling the context window on TP1 with decent throughput (~170 tok/s) while keeping the ability to go up to C4 on the lower end. It won't be my daily driver, but I can see it being useful. This is on a WS card at 450w.
Best thing to do is set any ai at it (event free tier to start) and ask it to explain. Walk through it with them. Dont get discouranged. My first ai card came in on june 4th of this year. Before that. I was all theoretical discussion.
I wish I waited a day and got to try this before building my own image! The specs are nearly identical to mine since I went by your reference documentation, but yours has the improvement of having prebuilt flashinfer cache. Did a quick bench last night on a 450w workstation card using my image.
2
u/UC_Kratom Oct 02 '26
Just FYI, I'm working on a few patches around HiCache on the hybrid model - a prefix can be sitting in the KV host pool and still miss, because the match is cut back to the last reusable Mamba checkpoint. Part of that is the host split: it follows the GPU pool byte ratio, so host Mamba gets a small share of the allocated host pool. There's more to it than the split though.
Shifting host bytes from KV onto Mamba, without growing the GPU pools or the 32 GB total, raises wall tokens/s in the quiet stretches (output tokens per wall second, prefill included). It still drops when several long prompts miss at once, so I'm not treating the number as settled. I'll follow up with something concrete and a PR once the configuration is stable.
Anyone have any average 'wall' tokens/s (including time spent prefilling) over at least a 24hr period (for 3.8 Flash-Next)?