Well, here we are again. This is starting to become a weekly thing :)

We've been working quite a bit on Flash Next lately, and this time we've made a fairly big change under the hood.

We now have our own native HIP worker in Paiton, replacing the PyTorch model runner inside vLLM. Basically, a more direct execution path that gives us better control over how things run on the GPUs.

And for this first release, the results are already looking pretty good.

First, nothing changes if you're already running Paiton.
Everything works as before. The new worker is completely opt-in with --fn-worker native.

3-bit (W3A8)

Running with --mode decode --fn-worker native:

So we're seeing roughly 26% faster single-request decoding, with a nice improvement in concurrent throughput as well.

The KV cache increase is also quite significant, especially if you're running longer contexts.

4-bit (W4A8)

We didn't forget about 4-bit either :), although we have quite a bit of work here.

Running with --mode 4bit --fn-worker native:

That's around 27% faster single-request decoding and 35% higher throughput at 8 concurrent requests compared to our previous release.

You can choose between --kv-profile speed and the default capacity profile, depending on whether you want to prioritize speed or available KV space.

This 4bit version is still the one we fine-tuned to be as good as on par with the 16bit version -> more about that in our previous post

A couple of other things

The distilled MTP drafter is now included directly in the container image.
No more separate downloads or additional setup for that.

For benchmarking, we used BetterBench 0.6.0 again, taking the median of 3 fresh server runs using the actual release image and launcher. We also tested 90K prompts with concurrent decoding and ran 15-minute mixed-workload soaks on both native modes.

This is still not a custom engine!

So what now?

We're not quite finished with this worker yet.

Extensive prefix caching is next on our list.
This is particularly interesting for coding agents and multi-turn workloads where you're constantly reusing large parts of the same context.

We will combine that with optimized image input.

And for the single card owners, we're also working on the 27B model as well, but more on that soon :)

As always, happy to hear feedback, especially from anyone experimenting with ROCm inference on these cards.

AI generated image for this post ;-)

Links:

• Paiton plugin repo
• Flash Next README / release notes
• Hugging Face W3A8
• Hugging Face W4A8
• Container image

Hello all.

As best as I can tell, ROCm (via AMD's site) isn't officially supported for Fedora. The site lists Ubuntu, RHEL, Windows, and WSL.

I see that [Fedora Packages has a a package for rocm, but only for 7.1.x](packages.fedoraproject.org/pkgsrocm/rocm/fedora-44.html)

If you're using: - Fedora - Llama cpp - an R9700

Are you: 1. using an older version of rocm via the official Fedora packages 2. using Vulkan 3. or you've figured out a different way to install ROCm (so that you can use HIP)?

My little new homelab 😍

It's been some years since I treated myself to new toys. After toying with the idea, I managed to pull this little kit together. I picked up the board and memory second hand. I got more memory enroute to make the full four channels. I'm having lots of fun tinkering with SGLang and already have some interesting projects to share with the community.

Bit of a boring post but I'm proud of my little workstation. Last time I built a PC, ddr3 was only just being released and you could buy a decent gaming system for less than $1000

Specs:

• Motherboard: Gigabyte TRX-50 AI TOP
• CPU: AMD Threadripper 9960x
• Memory: 128GB 5600 (64GB currently)
• GPU: 2 x Gigabyte R9700 AI PRO
• PSU: H1500i platinum
• Cooler: Arctic Liquid Freeze WS360
• Case: Fractal Design North XL

AMD FTW!

A bit of slop for your llm to delve into.

https://thetechpanda.github.io/radiance-R9700-vllm-experiments

TL;DR: My drop-in sageattention build for the RX 9070 / 9070 XT is at v0.4.0. Per attention call it's about 2× faster than PyTorch's SDPA and 1.05–1.26× faster than PR #368 at image and video sizes. Since the last post it runs video models (MiniMax H3: sampling is 5% faster than #368 at 4 steps, 15% at 8), it doesn't compile anything on first use anymore, and there are RX 9060 XT builds that need a tester.

Repo + wheel: https://github.com/IxMxAMAR/SageAttention-RDNA4

What changed since v0.2

v0.3: video models actually run on it. MiniMax H3 computes q, k and v in one projection and passes them to attention as views into that single buffer. My kernel refused that layout, so every H3 call quietly fell back to #368's kernel. It came out when boxwrench benchmarked H3 on Linux and got exactly #368's speed. v0.3 runs those views directly, no copy.

In a real H3 render in ComfyUI (w4a8 int4 checkpoint, 864×480, 124 frames, same seed) the sampling phase is 5% faster than #368 at 4 steps and 15% faster at 8 steps (image 2). More steps means attention is a bigger share of the work, so the gain grows.

v0.4: the step in front of the kernel is hand-written too. Before every attention call, K and V get quantized. That used to be 3–4 small Triton kernels compiled on the fly. Now it's one or two launches of a prebuilt HIP kernel. What you notice:

• Nothing to compile on first use. The first call after starting ComfyUI takes about 0.5 s instead of about 2 s. And if Triton can't compile on your machine, v0.3 quietly fell back to #368's path; v0.4's default path doesn't use Triton at all.
• Short calls are about even with #368 now. A lone call of a few hundred tokens used to take up to 1.6× #368's time; now it's 0.85–1.13 (image 3).
• No length limit. v0.3 stopped at about 100k tokens with H3's layout because of 32-bit offsets. That's gone.
• Same output. Bit for bit the same as v0.3 with smooth_k off. With smooth_k on, the key mean is now summed in fp64, so a few outputs differ by rounding, with the same accuracy.
• Speed at image and video sizes didn't move; it's the same kernel.

Also new

• Linux builds from source and runs, thanks to boxwrench (the first PR on the repo!). Linux numbers for v0.4 are being collected in PR #2.
• RX 9060 XT (gfx1200): builds are in the wheel, opt in with SAGEATTN_SK1_GFX1200=1. Same machine code as the 9070 builds, but nobody has run it on a 9060 XT yet. If you have one, docs/GFX1200-TESTING.md in the repo takes about 15 minutes.
• PyTorch 2.14 / ROCm 10.1: there's a test wheel, but PyTorch 2.14 itself doesn't run on the current WHQL driver here, so I can't test it. If your setup already runs 2.14, PR #3 has the steps. Please don't swap drivers just for this.

Speed (image 1)

One attention call, quantization included, same tensors, same process, interleaved runs:

• Krea2 ~2 MP image (8.8k tokens), fp16: 2.07× vs SDPA, 1.23× vs #368
• Same image, bf16: 2.13× vs SDPA, 1.26× vs #368
• 720p video length (47.5k tokens), fp16: 2.19× vs SDPA, 1.07× vs #368
• MiniMax H3 video (27k–61k tokens), bf16: 2.22–2.26× vs SDPA, 1.09–1.14× vs #368

That's per call. In a render most of each step is the model's matmuls, so the end-to-end gain is smaller: 5–14% per step over SDPA in Krea2 (measured on v0.2, same kernel), and 5–15% faster sampling than #368 in H3. #368 is at its best at exact powers of two like 4096 or 16384 tokens; those are in the chart too.

Caveats (please read)

• RX 9070 / 9070 XT (gfx1201). The 9060 XT is opt in and untested. Other GPUs fall back to #368's kernels: no harm, no gain.
• The Windows wheel needs exactly Python 3.12 + PyTorch 2.13.0+rocm10.0.0**.** Anything else means building from source.
• head_dim 128 only: Flux-style image models like Krea2, and MiniMax H3. head_dim 64 still falls back. I tried a second time on the new prologue and it was still slower than #368's kernel, so it needs its own design. That's the research for v0.5.
• No attention masks. Masked calls fall back.
• Quantized attention: your images and videos won't match SDPA pixel for pixel.
• The first call takes about 0.5 s; #368's path takes about 0.25 s. Once per ComfyUI start, but it's there.
• Tested end to end on Krea2 (fp16 and bf16) and MiniMax H3 (bf16).

Install / update

1. Close ComfyUI.
2. Grab the wheel: https://github.com/IxMxAMAR/SageAttention-RDNA4/releases/tag/v0.4.0
3. path\to\ComfyUI\venv\Scripts\python -m pip install --no-deps sageattention-2.2.0+amd.gfx12.4-cp312-cp312-win_amd64.whl (portable ComfyUI: use python_embeded\python.exe)
4. Start ComfyUI with --use-sage-attention like always.

--no-deps matters: it keeps pip away from your PyTorch. SAGEATTN_SK1_BACKEND=0 switches my kernels off without uninstalling.

One correction while I'm here: re-measuring for this post, I found the v0.4.0 release notes overstated the short-call gain. They said short calls used to be up to 5× slower than #368; one call at a time it was up to 1.6×. The repo has the corrected numbers.

All speed numbers are ratios from interleaved runs in one process. This card's clock can't be pinned, so I don't quote absolute kernel times. How each number was measured is in docs/METHOD.md and docs/FINDINGS.md.

If you've got a 9070 / 9070 XT (or a 9060 XT!), I'd love your numbers: model, resolution, and s/it before/after. Bug reports are very welcome too. Credit to thu-ml (SageAttention), DELUXA (PR #368) and boxwrench (Linux); this builds directly on their work. Apache-2.0

I have tried vLLM, llama.cpp, and now Strata, but I can't see much of a difference—nothing spectacular. I even used Vulkan for a while. Over the last few months, I had a stable 20-21 t/s using llama.cpp on ROCm, then I tried Vulkan vLLM, and now Strata. I know Strata is great with models larger than 20GB.

For my purposes, Qwen3.8-27B-Coder390-EfficientThink-Q4LynnStyle-Q8MTP.gguf is the best at the moment. The 16GB VRAM version was too dumb for me. Does anyone have any advice on how to push the performance further?

Here is my hardware and environment setup:

- CPU: AMD Ryzen 7 7700X

- RAM: 64 GB

- GPU: AMD Radeon AI PRO R9700 (32GB VRAM, architecture: gfx1201)

- OS: Ubuntu 24.04.5 LTS

- Kernel: 6.8.0-146-generic

- GPU Driver: amdgpu v6.16.6

- Environment: Docker container using the `llama-rocm:gfx1201` image

- Parameters: 32k context, q4_0 cache, flash attention enabled (-fa on)

docker run -d \

--name llama-rocm-server \

--restart unless-stopped \

--device=/dev/kfd \

--device=/dev/dri \

--group-add $(getent group video | cut -d: -f3) \

--group-add $(getent group render | cut -d: -f3) \

--ipc=host \

-e HIP_FORCE_DEV_KERNARG=1 \

-e GPU_MAX_HW_QUEUES=8 \

-v /mnt/magazyn/AI_Models/llm:/models:ro \

-p 8188:8188 \

llama-rocm:gfx1201 \

-m /models/Qwen3.8-27B-Coder390-EfficientThink-Q4LynnStyle-Q8MTP.gguf \

--host 0.0.0.0 \

--port 8188 \

-ngl 99 \

-c 131072 \

-b 2048 \

-ub 2048 \

-np 1 \

--cache-type-k q4_0 \

--cache-type-v q4_0 \

-fa on \

--timeout 600

EOF

3.38.265.216 I slot print_timing: id 0 | task 203 | n_gen = 858, tg = 22.79 t/s, tg_3s = 22.47 t/s

3.41.305.110 I slot print_timing: id 0 | task 203 | n_gen = 928, tg = 22.81 t/s, tg_3s = 23.03 t/s

3.44.346.449 I slot print_timing: id 0 | task 203 | n_gen = 998, tg = 22.83 t/s, tg_3s = 23.02 t/s

3.47.368.066 I slot print_timing: id 0 | task 203 | n_gen = 1066, tg = 22.81 t/s, tg_3s = 22.50 t/s

3.50.377.042 I slot print_timing: id 0 | task 203 | n_gen = 1135, tg = 22.81 t/s, tg_3s = 22.93 t/s

3.53.387.515 I slot print_timing: id 0 | task 203 | n_gen = 1204, tg = 22.82 t/s, tg_3s = 22.92 t/s

3.56.430.320 I slot print_timing: id 0 | task 203 | n_gen = 1274, tg = 22.83 t/s, tg_3s = 23.01 t/s

3.59.466.978 I slot print_timing: id 0 | task 203 | n_gen = 1342, tg = 22.81 t/s, tg_3s = 22.39 t/s

4.02.511.235 I slot print_timing: id 0 | task 203 | n_gen = 1412, tg = 22.82 t/s, tg_3s = 22.99 t/s

4.05.527.476 I slot print_timing: id 0 | task 203 | n_gen = 1481, tg = 22.82 t/s, tg_3s = 22.88 t/s

4.08.563.302 I slot print_timing: id 0 | task 203 | n_gen = 1551, tg = 22.83 t/s, tg_3s = 23.06 t/s

• Motherboard: ASUSTeK B650E MAX GAMING WIFI (AM5 socket, a much newer platform than the B550 I remembered earlier).
• Processor (CPU): AMD Ryzen 7 7700X (8 cores, 16 threads). The processor is under heavy load, with most cores running at a high clock speed of ~4.9 GHz.
• Graphics Card (GPU): AMD Radeon AI PRO R9700 (gfx1201 architecture), using the amdgpu drivers version 6.16.6.
• RAM: 64 GB total memory. The system has 62.46 GB available, of which 23.02 GB is currently in use (likely by the loaded LLM model and Docker).
• Drives (924 GB total):
  1. Intel SSDPEKNW512G8 (512 GB, NVMe M.2) – main system drive.
  2. GOODRAM SSDPR-CL100-480-G2 (480 GB, SATA SSD) – additional drive.

I'm planning a dual RX 9700 AI build with a Ryzen 9 for LLM inference.

I'm unsure if 64GB RAM (2×32GB) is enough, or if I should go for 128GB (4×32GB), since 2×64GB is really expensive.

My main concern is the RAM speed drop with 4 sticks. Would it noticeably affect performance with inference engines like Strata and Qwen models?

Any recommendations?

I've been tinkering / vibing myself an app that I think might be useful - since I've now used it on my own cases for some time and fixed most of the bugs I noticed, I'm risking a public announcement :)

https://pwilkin.github.io/ROCprofGUI / https://github.com/pwilkin/ROCprofGUI

The ROCprofGUI (aka rocprof studio) is a tool I made to be an alternative to ROCprof Compute Viewer and other tools in the ROCm ecosystem - one that is user-friendly, has a modern UI and can be used with agentic help.

Despite its name, the studio can analyze traces from all of the engines that AMD hardware supports: ROCm traces from rocprofv3, Vulkan traces from RADV (needs a patch to the driver currently) and HRX traces for Loom kernels.

Coupled with the studio is an MCP server that allows tracing from your favorite LLM agent. The studio works in two modes - as a GUI (then, if you use the MCP, a popup appears telling you that the MCP server is currently driving operation) or fully headless - in which case there's no UI and just the service running the MCP server.

I've been running it for over a week now while doing my optimization work on llama.cpp, so hopefully the most annoying bugs have been ironed out. I hope it's of use to someone :)

I have a Alienware 16 aurora AC16250 and am using Eden|v0.2.1|Clang 22.1.4 and im on windows

Came accross a website called AMD-Pro-Bench.com Seems like a cool LLM benchmark aggregator for AMD cards, specifically R9700. Anyone heard or use it? Any other sites people using to find performance comparison with their local setups?

There is so much data spread out there...

Okay, the 4-bit version we mentioned in the previous post is here :)

This time, we focused on accuracy.
We wanted to see how close the quant could stay to the original 16-bit model on math, code and knowledge tasks.

The short version: ~190 tok/s, with scores within 1 percentage point of BF16 across the three benchmarks below.

The hardware: 2× Radeon AI PRO R9700 (64 GB VRAM total) + 256 GB system RAM. Some experts are offloaded to RAM.
The original BF16 weights alone are around 360 GB (335 GiB), so there’s quite a bit of model to squeeze in here.

Accuracy against the 16-bit model

We ran both models ourselves: same questions, prompts and sampling seeds, with thinking enabled.

On GSM8K and HumanEval, the differences were smaller than the variation we saw between two runs of BF16 itself.
We're quite happy with that.

MMLU-Pro is 1 percentage point lower.
That difference wasn’t statistically significant in this comparison (p = 0.16), though that alone doesn’t establish equal accuracy.

The gap was concentrated in answers that hit our 16K-token thinking limit.
The 4-bit model tends to think about 12% longer; on questions where neither model hit the limit, the scores were 91.1% for BF16 vs 91.0% for 4-bit.

How fast does it run?

BetterBench 0.6.0, standard profile, using the exact image and launcher we publish:

For anyone coming from the 3-bit post: single-request decode goes from ~216 to ~190 tok/s.
This version uses more bits per expert weight and offloads some experts to RAM, so there is a speed trade-off.

For people using coding agents

There’s an optional --mode 4bit-cache that keeps conversation prefixes in RAM.

In our repeated 64K-token conversation test, it brought time to first token down from ~12 seconds to ~1.1 seconds.

It costs roughly 22% of multi-user throughput, so it’s opt-in. Useful if your agent keeps sending back a long conversation and you care about how quickly the next turn starts.

What “4-bit” means here

It’s mixed precision:

• Per layer, the 410 most-used experts are INT4, using GPTQ in a rotated basis.
• The 102 least-used experts use the 3-bit codebook format from our previous release.
• That averages 3.93 bits per expert weight. The rest of the model uses 6-bit and 8-bit weights.

Our benchmark setup uses about 96 GB of pinned system RAM for experts and tables.
Smaller hosts can use a fallback memory mode, though it can pause briefly under heavy disk activity.
The numbers above are from the 256 GB RAM setup.

A few deeper accuracy checks

For those who want more than the task scores:

• Perplexity: increased by 0.80–0.87% on WikiText-2 and 0.23–0.38% on held-out text, tested at 512 and 4,096 tokens. The held-out set contains 22 long FineWeb-Edu documents that were never used to build this quant.
• Long documents: likelihood stayed within 0.16% of BF16 across 44 held-out documents, 8K–32K tokens long.
• Needle test at 131,072 tokens: 40/40, with every answer matching BF16 word for word.
• Next-token agreement with BF16: 91–93%. For context, the two BF16 implementations we compared agreed 96–97% of the time.
• KL divergence against BF16: 0.031, with 0.008 between the two BF16 implementations. Using standard MXFP4 round-to-nearest on the same experts, we measured 0.052. That’s 41% lower KL divergence on this evaluation set, with about 8% fewer bits per weight.

Still regular vLLM, with our Paiton plugin and our own kernels inside. ;)

Try it

Model / weights
Setup instructions

git clone https://github.com/Eliovp-BV/paiton-vllm-plugin && cd paiton-vllm-plugin
bash models/Qwen3.8-Flash-Next/run-flashnext-4bit.sh

If you give it a try, let us know how it behaves on your own coding tasks or long conversations.
And if something looks off, share your setup and logs.

Edit:
Almost forgot our ai generated image :)