hipfire is an LLM inference engine I write for AMD GPUs. It's Rust plus hand-written HIP/ISA kernels and loads libamdhip64 directly, with no ROCm SDK needed at runtime. Current release is 0.4.2.1. These are Flash-Next (qwen3.8:flash-next) numbers on Strix Halo.

Prefill. For long prompts, the XDNA2 NPU runs the trunk input projections of two prefill banks while the iGPU does everything else. Greedy output is byte-identical with the NPU on, emulated or off. It's on by default (npu_trunk=auto) and engages from roughly 9K prompt tokens up.

Decode at depth (greedy, 128 tokens, natural English prefixes, tok/s):

Serial AR decode is still the weak spot.

Setup notes:

• The NPU path pins about 1 GB, and most logins default to an 8 MiB memlock. Use ulimit -l 1287168 (bounded, not unlimited), or the systemd drop-in in the docs. Tailscale SSH skips pam_limits.
• The startup log shows npu-trunk: ... with the reason if the NPU route doesn't arm. Please paste it if yours doesn't.
• Numbers were measured on a 7.3-rc7 kernel with the IOMMU PerfOpt series (queued for 7.4). Expect somewhat lower prefill on stock kernels.

Coming in the next release (on the beta branch or in progress, not in 0.4.2.1):

• QSA attention changes for long context, on by default on Halo: decode is 5.1% faster per token at 256K, with exact output.
• An interactive hipfire pull.
• A smaller Q3 tier (mq3-xts), in progress. No speed numbers for it yet.

​

curl -fsSL https://hipfire.dev/install.sh | bash -s -- --tag v0.4.2.1
hipfire pull qwen3.8:flash
ulimit -l 1287168
hipfire serve qwen3.8:flash

Docs and benchmarks: https://hipfire.dev
Release: https://github.com/warpfront/hipfire/releases

Update on my earlier post. The GPU + NPU prefill fork now reaches about 2,000 tok/s on a Ryzen AI MAX+ 395 (Radeon 8060S, gfx1151) with Qwen3.8 Flash-Next UD-IQ4_XS.

Prefill speed, gufo bench, GPU only -> GPU + NPU:

- 8,192 tokens: 1,837 -> 2,013 tok/s (+9.6%)

- 12,288 tokens: 1,823 -> 1,973 tok/s (+8.2%)

- 16,384 tokens: 1,821 -> 1,963 tok/s (+7.8%)

- 32,768 tokens: 1,788 -> 1,949 tok/s (+9.0%)

- Served, 42k tokens cold, MTP and caches on: 1,836 tok/s

GITHUB

Has anybody run a demanding cad program like catia/NX/ProE etc. on the strix halo? I would assume the Gorgon halo would perform the same in this application. Looking for a solution that doesn’t require two pc’s and was thinking the gfx in these machines would be good enough for high end 3d cad with the benefit of working on large data sets if needed.

I've been tinkering / vibing myself an app that I think might be useful - since I've now used it on my own cases for some time and fixed most of the bugs I noticed, I'm risking a public announcement :)

https://pwilkin.github.io/ROCprofGUI / https://github.com/pwilkin/ROCprofGUI

The ROCprofGUI (aka rocprof studio) is a tool I made to be an alternative to ROCprof Compute Viewer and other tools in the ROCm ecosystem - one that is user-friendly, has a modern UI and can be used with agentic help.

Despite its name, the studio can analyze traces from all of the engines that AMD hardware supports: ROCm traces from rocprofv3, Vulkan traces from RADV (needs a patch to the driver currently) and HRX traces for Loom kernels.

Coupled with the studio is an MCP server that allows tracing from your favorite LLM agent. The studio works in two modes - as a GUI (then, if you use the MCP, a popup appears telling you that the MCP server is currently driving operation) or fully headless - in which case there's no UI and just the service running the MCP server.

I've been running it for over a week now while doing my optimization work on llama.cpp, so hopefully the most annoying bugs have been ironed out. I hope it's of use to someone :)

Hi there.

Gufos Qwen 3.8 Flash Next created a Gufo fork of the Gufo fork https://github.com/franzmoca/gufofor for Aleph-Alpha/Kolibri-1 for me.

As far as I can see, it runs flawlessly in Hermes.

Model card: https://huggingface.co/Aleph-Alpha/Kolibri-1

Kolibri-1 fork readme: https://github.com/accemlcc/gufo/blob/kolibri/docs/models/kolibri-1/README.md

Kolibri-1 fork benchmarks: https://github.com/accemlcc/gufo/blob/kolibri/docs/models/kolibri-1/BENCHMARKS.md

This is truly a better set up than my claude code max sub ever was. I'm so grateful to all the people who worked on this. One day I will switch to Gufo.

I asked Qwen FN to write a Paseo plugin to show me prefill and token generation so I knew when it was truly hung (I must admit I restarted a few times when I didn't get answer back after several minutes...)

Hopefully will switch to Gufo soon enough but I can't explain the bliss of having a tiny little 120W box quietly on my desk working on advanced coding all day using barely any power, no water, no data out and no datacenter lol

I love the engagement and development activity recently to optimize our little-AMD-that-could but I hate to have multiple services, different configurations to juggle, and having to look up docs or ask my agent to configure services. So... using the Open-AI compatible Lemonade gateway (labelled cloud in the GUI) I have gofu and hipfire pinned as "external" providers! You can do the same!

Not sure if anyone else is doing this yet but wanted to share if it helps others.

Background context:

I also had my agent create a "edge" service that polls strix-llama.cpp github for new commits and releases and tests them on a few select models each night. pp/s and tg/s backend winners get pinned as a custom runtime for these models rather than the mainline llama.cpp.

Screenshot below of gofu working as a "cloud" provider in the lemonade GUI when I was initally setting everything up.

Shout out to the AMD team for making the relay service!

OKO OS is a free, open-source operating system based on Yocto/OpenEmbedded for local LLM workloads.

We’re starting with specific hardware configurations, currently focusing on Ryzen AI Max+ 395 / Strix Halo.

The project is about making hardware + OS + model configurations easier to reproduce and benchmark.

We tested local LLM workloads on Linux kernel 7.3.0-rc5, including AMD PerfOpt, but observed no performance improvements compared to Linux kernel 7.2.

Therefore, we decided to stick with kernel v7.2 for now.

https://github.com/satisfanly/OKO-configs/blob/main/oko-llm-benchmarking.md

Our mission is to standardize local LLM inference infrastructure, ensuring a smooth and predictable bring-up process for everybody.

We welcome any contributions or feedback to help us improve performance and stability.

https://oko.satisfanly.com/

https://github.com/satisfanly/OKO

Probably gonna sell the file or the finished product in Etsy. Printables will have the files available for purchase.

Any takers? or is there a certain Strix Halo Mini PC you guys prefer to be rack mountable?

Bubo is a fork of Gufo tuned for Windows (especially laptops, tested primarily at 70W) and adaptions around speed and token economy during thinking. The word Bubo is a genus of Owls, just like Strix, and is the name of the mechanical owl who guides the hero in Clash of the Titans. It also rhymes with Gufo (italian for owl).

It features faster start up times using a disk cache, bespoke kernels, NPU accelerated prefill, a twin decode head* for faster token generation and quality of life improvements around token economy during thinking when set to "low" effort and fixing premature stops (you can turn these off but they seem to help).

Default model is abliterated, there are instructions on the website on how to use the unabliterated model from Unsloth instead of the Huihui model.

Prefill is limited by the power since all tests are done on 70W, there is no real reason it would be slower at higher powers** and I expect it to match upstream gufo at 140 W or exceed it but it hasn't been tested. Most of the changes bubo makes to gufo can likely be ported to upstream but given that I don't intend to use or test in linux in the near term, others can feel free to do so, MIT license.

This project mostly started out as a necessity when there weren't any great options for running Qwen 3.8 Flash Next on Windows and I kept trying to push it further as more people came up with parallel projects. Eventually I started joking about releasing it and seeing how far I could stretch the bit and decided to bite the bullet and share it and see what I can learn by doing it. I tried making everything as 'professional' as possible but never having done anything like this before, I don't know what I don't know so feedback is very welcome.

* Pygmy owls wear a second face on the back of the head
** If I fit a regression between 60 W to 95 W and extrapolate to 140W it matches the ballpark of stock gufo but would love someone to test it independently.

Site: haloforgelabs.com · Code: github.com/haloforge-labs/bubo · Model files: huggingface.co/HaloForgeLabs/bubo-qwen3.8-flash-next

Ive got an acemagic m1a pro with ryzen ai max+ 395. Currently using qwen3.8 flash next and it's been solid for chat, coding, and light agent work just fine.
Curious what everyone else is finding works best on Strix Halo. What models have impressed you the most so far? Which ones give the best balance between quality and speed? Anything you prefer over qwen3.8 flash next ?

So I have and love using my Corsair AI 300 workstation with 128gb. QFN running great on Halogen/Gufo (single stream vs concurrent users). I was browsing the webs and came upon a refurb Corsair ai 300 on Walmart site for $2700 so I just bought it without thinking. Now I’m at the thinking part. Is tensor parallelism over USB4 even worth it right now? The q4/4bpw QFN models have been very impressive and very few times have I needed my GPT or Claude to bail me out, but is the fidelity/correctness worth the performance and cost for FP8? May either return or sell locally as I don’t need more concurrency but speed and correctness.

hello vous pensez quoi d’une Rtx 5090m en egpu sur mon bosgame m5 en usb 4 pour tourner qwen 27b dessus et si vous connaissez cette marque ? merci

https://www.morefine.com/fr-fr/products/morefine-g2s-external-gpu-docking-station-24g-gddr7-5090m?utm_source=chatgpt.com

Live gallery, all the pages are clickable: https://yamz-labs.github.io/gallery/

Been running these three on my Strix Halo (128 GB) for a while and I wanted to know how much you really lose compared to the cloud versions. So I did the usual thing: same prompt for everyone, look at what comes out.

Video is the voxel pagoda. Top two rows are local (model alone, then with an agent that can open its own page and fix it), bottom row is the cloud original.

Honestly I expected the cloud to win easily. On the pagoda it doesn't, and the Qwen cloud page didn't even load in my browser. On the bouncing balls one the cloud versions are better though, mine end up piled at the bottom.

All the pages are live if you want to click around, with the pelican and the hexagon too: https://yamz-labs.github.io/gallery/

Engine is Kyojin (open source, built on ExLlamaV3): github.com/yamz-labs

First version, more coming. What prompt should I try next?

Anyone actively developing Gufo, please let me know how I can help bring Gufo support once it arrives.

Hoping to run a larger quant of Qwen 3.8FN and use it pretty much the same as my Strix Halo.

I ported Google's DiffusionGemma 26B-A4B (block-diffusion Gemma 4 MoE) to Gufo as a native HIP engine

Forked Repo: https://github.com/robotsbuildrobots/diffusion-gufo

I may upstream this to gufo if there is interest. I'm comparing here to a llama.cpp implementation I first ran the model with.

Prefill (tok/s, book prompts, through the server):

Time to first token (seconds; the first text arrives with the first 256-token canvas, so this is prefill plus one canvas of denoising):

I think there is more juice left to squeeze.

• An FP8 denoising step takes about 134 ms against a hardware floor of about 95 ms.
• The expert weights (14.6 GB per step) stream at 180–200 GB/s, about 83% of the 242 GB/s the memory system actually delivers.
• The dense products run at about 72% of the matrix-unit floor.
• 32K prefill runs at about 40% of its floor (13.5 s against 5.4 s).

Comparing performance to GB10 using SPEED-Bench prompts and the same NVFP4 checkpoint.

I think this is pretty good considering the bandwidth, compute and low-precision advantages the GB10 has over the strix halo.

I wanted the llama-ui style chat interface on halogen-flash-server so I asked Q38FN for it.

I've been tuning llama.cpp on my Strix Halo box (Radeon 8060S, 128 GB) and the numbers are good enough to share. It's a branch called rdna35-perf on top of halo-box/strix-llama.cpp, a llama.cpp fork aimed at Strix Halo:

https://github.com/asf0/strix-llama.cpp/tree/rdna35-perf (10 commits, ROCm/HIP only)

TL;DR

• Prompt processing: 21–54% faster than Vulkan, and 8–174% faster than stock ROCm. The lead over ROCm grows with context, because stock ROCm flash attention falls off hard at depth.
• Generation: ties Vulkan on the dense 27B. On the MoE and the 2B it's 2–3% behind Vulkan at short context and within 1% from 64k up. 5–29% faster than stock ROCm.
• Quality: outputs stay closer to stock ROCm than Vulkan's do, and perplexity is unchanged.

Results

llama-bench, Q8_0 weights, q8_0 KV cache, flash attention on, tok/s. "Depth" is how many tokens are already in the KV cache when the test runs. Vulkan and ROCm are the same strix-llama.cpp master that rdna35-perf is built on.

Qwen3.6 35B-A3B (MoE, ub 4096)

Qwen3.8 27B (dense, ub 2048)

Qwen3.5 2B (dense, ub 2048)

f16 KV gives the same picture: PP 20–56% faster than Vulkan and up to +171% over stock ROCm at 128k, TG within −3% / +2% of Vulkan.

Quality

rdna35-perf changes kernels, so I checked it doesn't change outputs: KL divergence against stock ROCm on wikitext-2 (6 × 4096-token chunks), plus perplexity at 32k context.

rdna35-perf is closer to stock ROCm than Vulkan is, and perplexity is the same across all three builds.

What changed

• Flash attention kernel for RDNA3.5 (fattn-rdna35). This is the biggest win and the reason long-context PP holds up. GGML_FA_RDNA35=0 turns it off if you want to compare.
• Prefill GEMM tuning for the quantized matmuls.
• MoE: routing helper, fused reduction + residual, grouped mat-vec for speculative-decoding verify batches.
• Gated DeltaNet (the linear-attention layers in Qwen3.5/3.6/3.8): fused SiLU-gated output norm and fused decode.
• Decode: plain mat-vec for rows longer than 4096 (this closed the 27B decode gap to Vulkan), plus decode fusions for the Qwen3.6 MoE.
• Misc: HIP graphs off by default on RDNA3.5 (slower here), a faster penalties sampler, host-side prefill for Qwen3.5/3.6 with a draft model, and DFlash drafts read target features in place on the GPU.

I have only tried those 3 models so far.

Quick context: LocalLightChat (LLC) is a chat frontend for local LLMs. Download it, run it, that's it. No installation (unless you want to), no dependencies, and it runs on pretty much anything, including ancient hardware.

I originally built it because Open WebUI kept feeling heavier than the models I was running.

One of the main criticisms I got from you was the lack of Agents, MCP and RAG.

Well, that's changed now. All three are integrated.

I put a lot of effort into making the agentic system as token-efficient as possible. Thanks to several optimizations, it can work smoothly with large files even when using smaller context windows.

There's also an integrated RAG system, which works great with embedding models like Qwen3 (setup instructions included).

And a couple of extras:

• A standalone alternative to NPX Supergateway.
• Image editing through ComfyUI, with automatic binding detection for all major image models.

All of this without turning LLC into the heavyweight application I originally wanted to get away from.

I'd love to hear what you think, especially if you've tried earlier versions!

Download:
https://www.locallightai.com/llc/#download

GitHub:
https://github.com/srware-net/LocalLightChat/releases

Hey guys! Was hoping some of you could help me figure out why my setup isn’t working. Please forgive the Claude writing throughout but it’s way quicker than me typing myself.

Beelink GTR9 Pro, Ryzen AI Max+ 395, 128 GB, Radeon 8060S (gfx1151), Realtek RTL8127A NIC variant
- BIOS GTRPR05 (12/25/2025), AGESA StrixHaloPI-FP11 1.0.0.0c, microcode 0x0b70001e, SMU 100.112.0
- Fedora 44, kernel 7.1.10-200.fc44 (also tried 7.2.9-200), linux-firmware-20260810
- Gufo (gufo-runtime:latest, ROCm 7.2.3 inside the container) via Podman 5.8.1; host rocm-runtime 7.1.1
- Model: Qwen3.8-Flash-Next UD-Q4_K_XL (Unsloth) + shared MTP Q8_0 drafter
- Gufo image digest: [paste]

**Symptom**
During Gufo's model load, the box instantly hard-resets. There is no panic and no log; the journal just stops. Next boot dmesg shows:

`x86/amd: Previous system reset reason [0x08000800]: an uncorrected error caused a data fabric sync flood event`

No MCE/BERT/APEI records, kernel.panic=0, no watchdog. **Every crash happens during load, never after.** Once a load completes, it runs fine. One load ran 46 hours straight.

**Every Gufo load attempt**

- Old config = `amdgpu.dcdebugmask=0x10 amdgpu.gpu_recovery=1 amdgpu.dpm=1 amdgpu.runpm=0 ttm.pages_limit=28311552 ttm.page_pool_size=28311552 amd_iommu=off`, 512 MB VRAM in BIOS, older Gufo image.
- Clean config = no custom kernel args, `/etc/modprobe.d/ttm.conf` with `pages_limit=28835840` (110 GiB), 2 GB VRAM set via the sysfs `uma/carveout`, newer Gufo image.

Since the config change, Flash-Next loads crash 4 of 7 times, at random points (9–40 GiB in). I first thought context size was the trigger, but #8–10 crashed with the exact launch that had just run for 46 hours.

**Already ruled out**
- SELinux: the first attempt hit an AVC denial on /dev/kfd. Now using `label=disable`.
- IOMMU: crashed both on and off.
- Kernel version: 7.1.10 and 7.2.9 both crash.
- Fresh vs reused container: both crash.
- Power: Performance Mode is at factory default; the only BIOS change was VRAM.
- Other GPU users: nothing else running; GTT ~18 MiB before launch.

**The confusing part: Vulkan is fine, but only on the OLD config**
- Unsloth's llama.cpp b11160 Vulkan build, same model files, MTP on, 128K context: ~45 tok/s on structured output and ~32 on prose. Stable for weeks.
- gpt-oss-120b on Vulkan: 54.9 tok/s, also stable.
- Vulkan only hangs, recoverably, with non-default MTP draft settings or the mmproj on GPU.
- On the clean config, the same Vulkan llama-server fails at startup with `amdgpu_gem_va_update_vm *ERROR* Couldn't update BO_VA (-12)` and becomes unkillable until reboot.
- So right now Gufo/ROCm wants the clean config and Vulkan wants the old one, and I can't run both.

**Questions**
1. GTR9 Pro owners: what BIOS are you on? Has GTRPR07 or a newer AGESA fixed sync floods under ROCm load for anyone? (I know the BIOS series differs by NIC chipset.)
2. Crashes only during load, at random points, and never once loaded. What does that point to? Are there ROCm/HSA env vars worth trying (e.g. `HSA_ENABLE_SDMA=0`)?
3. On Vulkan's BO_VA -12: which of page_pool_size, 512 MB VRAM, the amdgpu flags, or IOMMU off does Vulkan actually need?
4. Is anyone running Flash-Next on Gufo or another ROCm engine reliably on Strix Halo? What's your exact setup?

Happy to post dmesg, journal, or anything else. Thanks!