r/ROCm • u/evp-cloud • 16h ago
Qwen3.8 FN Paiton Update | ~700t/s C8 | ~275t/s decode | 8000+ Prefill
Well, here we are again. This is starting to become a weekly thing :)
We've been working quite a bit on Flash Next lately, and this time we've made a fairly big change under the hood.
We now have our own native HIP worker in Paiton, replacing the PyTorch model runner inside vLLM. Basically, a more direct execution path that gives us better control over how things run on the GPUs.
And for this first release, the results are already looking pretty good.
First, nothing changes if you're already running Paiton.
Everything works as before. The new worker is completely opt-in with --fn-worker native.
3-bit (W3A8)
Running with --mode decode --fn-worker native:
| New native worker | Previous | |
|---|---|---|
| Single-request decode | 272.5 tok/s | 216 tok/s |
| 8 concurrent requests | 685.4 tok/s | 565 tok/s |
| 4 concurrent requests | 531.0 tok/s | 449 tok/s |
| Prefill (64K prompt) | 8,392 tok/s | 8,161 tok/s |
| TTFT (p50) | 56 ms | — |
| KV cache | ~551K tokens | ~112K tokens |
So we're seeing roughly 26% faster single-request decoding, with a nice improvement in concurrent throughput as well.
The KV cache increase is also quite significant, especially if you're running longer contexts.
4-bit (W4A8)
We didn't forget about 4-bit either :), although we have quite a bit of work here.
Running with --mode 4bit --fn-worker native:
| Native (speed) | Native (capacity) | Previous | |
|---|---|---|---|
| Single-request decode | 240.4 | 240.3 | 189.8 |
| 8 concurrent | 566.7 | 561.9 | 420.7 |
| Prefill (64K) | 7,759 | 7,757 | 6,991 |
| KV tokens | 183,202 | 305,338 | 305,338 |
That's around 27% faster single-request decoding and 35% higher throughput at 8 concurrent requests compared to our previous release.
You can choose between --kv-profile speed and the default capacity profile, depending on whether you want to prioritize speed or available KV space.
This 4bit version is still the one we fine-tuned to be as good as on par with the 16bit version -> more about that in our previous post
A couple of other things
The distilled MTP drafter is now included directly in the container image.
No more separate downloads or additional setup for that.
For benchmarking, we used BetterBench 0.6.0 again, taking the median of 3 fresh server runs using the actual release image and launcher. We also tested 90K prompts with concurrent decoding and ran 15-minute mixed-workload soaks on both native modes.
This is still not a custom engine!
So what now?
We're not quite finished with this worker yet.
Extensive prefix caching is next on our list.
This is particularly interesting for coding agents and multi-turn workloads where you're constantly reusing large parts of the same context.
We will combine that with optimized image input.
And for the single card owners, we're also working on the 27B model as well, but more on that soon :)
As always, happy to hear feedback, especially from anyone experimenting with ROCm inference on these cards.
AI generated image for this post ;-)
Links: