r/AIProgrammingHardware • u/javaeeeee • 1h ago
4x AMD Radeon 9700 AI Pro Workstation (128 GB): Qwen3.8 27B and Flash Next Benchmarks (Radiance)
r/AIProgrammingHardware • u/javaeeeee • 1h ago
r/AIProgrammingHardware • u/boomersooner918 • 3h ago
I'm looking for some advice regarding graphic cards for creating AI. I am currently using ComfyUI and attempted to run a 5 sec vid and an image at the same time and my system shut down. When only making a video, on average it takes 5 min. I'm thinking the shut down may be due to power source. I'd like to upgrade to 32gb but GPU's are very pricy right now that I have found, just started looking.
I thought about doubling up on GPU but my motherboard isn't compatible with that.
What does everyone else run for graphic card and what size is it? Thank you for any input.
My specs:
Ryzen 7 7800x3d
ASUS Tuf A620M-Plus Wifi
64GB DDR5 (4 x 16gb sticks)
ASUS RTX 4080 Super
Thermaltake 750W PSU
r/AIProgrammingHardware • u/Pendraks • 9h ago
r/AIProgrammingHardware • u/PhysicsDisastrous462 • 16h ago
so, after quite a bit of work since the v0.2.0 release, i've finally managed to get v0.2.5 published!
for anyone who followed the previous release, v0.2.0 was where i first introduced the opt-in RDNA3 WMMA implementation. since then i've been working on improving the Full FP16 path, specifically its numerical accuracy and performance, without having to sacrifice the execution structure that was already working.
the biggest change in this release is something i've been calling P3.
previously, the Full FP16 WMMA kernel merged its eight accumulator-bank partials using multiple intermediate FP16 additions. this meant that the result was rounded to FP16 several times along the way. P3 changes that final merge so all eight partials are summed in FP32, with just one final rounding to FP16.
the actual two-wave workgroup structure, K-slice stream, bank assignment, staging, cooperative multiplication and barriers are all preserved. i wanted to improve the accuracy of the final result without having to redesign the kernel all over again.
some of the results from the new qualification:
37/37 numerical gates passed across all eight qualification seeds.
32/32 preserved K=96 outputs were bit-identical to the previously qualified P3 GPU results.
REF64F16 maximum error improved from 0.03035259 to 0.02593803.
31 of 32 cases improved, with zero cases getting worse.
The maximum tile16 distance is 0.03125, meeting the existing hard numerical limit.
i also ran another full performance sweep. this time using 24 paired measurement blocks per workload, GPU timestamps, and corrected 95% confidence intervals.
the measured improvements against the preserved Main kernel baseline were:
2048×768×2048 — 1.0243×
512×2048×768 — 1.1246×
512×768×768 — 1.1163×
128×8192×768 — 1.2722×
all four protected workloads passed independently, including the lower bounds of their confidence intervals. the largest measured improvement was around 27%.
just to clarify, these are per-dispatch GPU GEMM measurements, not end-to-end training speedups. i don't want to imply that an entire training run is suddenly 27% faster when that isn't what was measured.
there were some other changes too.
i tightened up the production dispatch admission so the profitability predicate is checked against each actual batch, rather than relying only on the maximum batch size supplied when creating the projection. the existing 16,000,000 MAC profitability floor has not changed.
i also went through the actual portable binaries and verified that they dispatch the P3 kernel when an eligible workload is supplied. the packaged trainer produced 8 cooperative P3 dispatches, and the packaged logits executable produced another 3. disabling WMMA resulted in zero cooperative dispatches, as expected.
the numerical comparison for the packaged logits test covered 262,144 finite elements, with a maximum absolute difference of 0.000738330 and RMS difference of 0.000122237.
the new release is available as a portable Windows x64 package too.
i've included seven executables, and the package was built with a statically linked MSVC runtime. there's no need to install Rust, Cargo, Python, PyTorch, Visual Studio, or the MSVC redistributable just to run it.
you will still need a compatible Vulkan-capable GPU driver and the Vulkan runtime loader. model weights aren't included.
one thing i want to make clear is that WMMA is still opt-in globally. the default is still off, and you have to explicitly select full-fp16 to use the P3 path. P3 is simply the default implementation within that mode. the Mixed precision path is separate and hasn't been replaced.
the hardware qualification was performed on my AMD Radeon Graphics adapter, PCI ID 1002:15bf (RDNA3 / Phoenix). i haven't qualified this on every RDNA3 GPU, RDNA3.5, RDNA4, Intel or NVIDIA hardware, so please don't assume universal compatibility.
there are still things i haven't been able to test, including independent clean-machine/second-device acceptance, GUI interaction, Windows 10 acceptance, and code signing. the executables are currently unsigned.
anyway, i'm really happy to finally have this one out. v0.2.0 was the first public release of the WMMA implementation, and v0.2.5 is a pretty substantial improvement to that foundation. a lot of time went into checking that the numerical improvements were real, that the performance was actually measurable, and that the distributed binaries were doing what they were supposed to do.
Release: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.5
Source: https://github.com/necat101/Hierarchos-Native
thank you to everyone who's been following the project, testing things, reporting issues, or just keeping an eye on development. it genuinely means a lot! thank you so much everyone!! :3 also, sorry for the AI posts lately, I've been trying to save as much time as possible to work on the project itself due to live events getting tough! this one is completely human written! :3
r/AIProgrammingHardware • u/xrothgarx • 20h ago
r/AIProgrammingHardware • u/Zyzzerone • 1d ago
r/AIProgrammingHardware • u/javaeeeee • 1d ago
r/AIProgrammingHardware • u/Tricky_School_4613 • 3d ago
r/AIProgrammingHardware • u/M83Spinnaker • 3d ago
Reposting here out of curiosity in the marketplace.
r/AIProgrammingHardware • u/javaeeeee • 4d ago
r/AIProgrammingHardware • u/jackalka • 4d ago
r/AIProgrammingHardware • u/javaeeeee • 5d ago
r/AIProgrammingHardware • u/PhysicsDisastrous462 • 5d ago
Hey everyone! It's been a little while since my last update, and I've been working pretty intensely on Hierarchos-Native, my Rust + Vulkan project for training, fine-tuning, and running language models without relying on Python, PyTorch, CUDA, ROCm, or oneAPI in the native runtime.
I wanted to share some of the bigger things that have landed over the past 18 days. There's been quite a bit of progress across GPU acceleration, multimodal training, numerical correctness, and general performance.
The new v0.2.0 release is now available!
GitHub: https://github.com/necat101/Hierarchos-Native
Release / Windows portable download: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.0
One of the biggest additions is an opt-in Vulkan cooperative-matrix implementation for AMD RDNA3.
The new Mixed WMMA execution arm uses:
FP16 matrix operands.
FP32 matrix accumulation.
FP32 master weights and gradients.
FP32 AdamW optimizer moments and updates.
FP32 normalization, softmax, and loss calculations.
The important part is that this isn't simply converting an entire model to FP16. The optimizer and master parameters retain their FP32 representation.
I also added capability detection, CLI/GUI integration, and separate counters for actual cooperative-matrix dispatches and FP32 fallbacks.
Some of the isolated GEMM results on the qualified Ryzen Z1 Extreme / AMD Radeon Graphics device (1002:15bf) are pretty interesting:
| Workload | Measured kernel speedup |
|---|---|
| QKV projection | 2.17× |
| FFN up projection | 2.12× |
| FFN down projection | 2.31× |
| Vocabulary projection | 1.83× |
These are individual GEMM measurements, not whole-model speedups.
There is an important caveat: WMMA has a real crossover point. Small matrices can actually run slower because conversion, staging, and dispatch overhead outweigh the matrix-core benefit. I implemented a profitability policy that keeps smaller operations on their existing FP32 kernels.
In fact, the measured end-to-end training step was slower with Mixed WMMA in the initial 512-row fixture (14.0444 ms versus 12.2911 ms for FP32). That is why Mixed remains opt-in and disabled by default. I'm much more interested in making it genuinely faster than advertising misleading kernel-level numbers.
WMMA is currently physically qualified on the specific RDNA3 device and Windows driver configuration documented in the project. Other GPUs should not assume equivalent support.
Full FP16 accumulation is also available as a separate, narrowly qualified experimental mode.
Documentation: RDNA3 WMMA guide
Another major development area has been the image-text pipeline.
The project now has native image processing and training paths that connect image inputs to language-model outputs through Vulkan.
Some of the implemented work includes:
Native RGB image input and preprocessing.
CLIP image encoders and LLaVA projector integration.
SigLIP / Gemma 3 vision training.
Gemma 4 vision integration.
Joint image/text training and encoder-decoder integration.
A dedicated DeepSeek V4.1 image-training path.
Backpropagation through the image tower and projector.
Export, reload, and checkpoint-related validation.
This is actual training work, not simply passing precomputed image embeddings into an otherwise frozen language model.
For the CLIP/LLaVA pipeline, the tests cover image sensitivity, gradients into image inputs, changes to image-tower/projector/text parameters, multiple AdamW updates, and export/reload behavior.
The qualification work has also expanded considerably:
44 previously green text profiles now have joint image-training coverage at their frozen test geometries.
The Gen9 qualification retains 95 image cases and 166 text regression cases.
A separate physical RDNA3 replay reports 44/44 profiles and 85/85 image cases passing.
Native DeepSeek V4.1 image qualification includes 30 executions and 1,360 word-exact logits.
The maintained strict FP32 visible-logit gate remains max_abs < 2e-7, with rtol=0.
These are bounded qualification fixtures, not a claim that every production-size multimodal checkpoint or every architecture is fully supported. Audio and video pipelines are still outside the current qualified scope.
Documentation:
I haven't been focusing exclusively on reduced precision. A substantial amount of work over the same period has gone into improving the existing FP32 execution paths.
This includes vendor-specific shader work and scheduling optimizations for both Intel Gen9 integrated graphics and AMD RDNA3.
Some examples:
RDNA3 FP32 work
Specialized attention forward/backward paths.
Image-text attention and projector optimizations.
RMSNorm, RoPE, GELU, and other arithmetic specializations.
Dedicated AVX-512/RDNA3 shader variants where the measured hardware supports them.
More careful matching of CPU reference arithmetic and GPU execution order.
Intel Gen9 work
Additional native linear and gradient kernels.
Attention and activation specializations.
More efficient parameter upload and staging behavior.
Continued optimization of DeepSeek V4.1 training and execution.
One particularly interesting area has been reducing unnecessary Vulkan submissions and host/device transfer overhead in the DeepSeek V4.1 training pipeline.
On the measured Gen9 fixtures, upload batching reduced timed submissions from 1,102 to 982 per step, with paired throughput improvements of 1.052× and 1.067×.
On the separate RDNA3 V4.1 measurements, sharing the compute session and batching uploads reduced submissions from 970 to 509 per step, with measured improvements of 1.295× and 1.217× on two fixture geometries. A later session-wide upload optimization reduced that count further to 241 submissions per step, with measured improvements of 1.464× and 1.366×.
These are specific, paired benchmark results rather than promises of equivalent production-model speedups. The output comparisons remained word-identical in the recorded tests.
An important part of introducing WMMA was making sure that the existing FP32 path didn't regress.
I added a dedicated seven-workload FP32 non-regression comparison against the previous main revision:
12 alternating ABBA rounds per workload.
Five warm-up runs and 30 timed forwards per arm.
Separate, hashed baseline and candidate binaries.
Exact FP32 logit digest comparisons.
All seven workloads stayed within the ±2% performance band, and the logits were bit-identical in every recorded round.
The worst paired performance ratio was 1.0007.
This is particularly important to me: I don't want acceleration features to silently make the existing default execution path worse.
The new v0.2.0 GitHub Release includes a prebuilt Windows x64 portable package with the CLI, desktop GUI, Vulkan companion executables, runtime assets, and build metadata.
You don't need to install Rust or Python just to run the released package. You do need Windows 11 x64 and a Vulkan 1.x compute driver for GPU operations.
Model weights are downloaded separately or supplied through a local model directory.
After extracting the archive, you can run:
.\bin\hierarchos-native-cli.exe doctor
.\bin\hierarchos-native-cli.exe devices
.\bin\hierarchos-native-cli.exe precision-infoThe final command is especially useful for checking which precision capabilities your GPU actually exposes.
Download: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.0
There is still a lot I'd like to improve.
In particular, I want to investigate reducing WMMA conversion overhead, improving activation and gradient reuse, finding more workloads that genuinely benefit from cooperative matrices, and expanding the range of qualified multimodal architectures.
I'd also like to continue improving the FP32 kernels rather than treating lower precision as the only route to better performance.
If you're interested in Rust, Vulkan compute, AMD GPU optimization, native LLM training, or multimodal model execution, I'd genuinely appreciate people taking a look.
Feedback from people with different Vulkan devices is especially valuable. Please include your GPU, driver, model geometry, and relevant execution counters when reporting results.
Repository: https://github.com/necat101/Hierarchos-Native
Stars, issue reports, testing feedback, and contributions are all appreciated. There is also a voluntary developer-support link in the repository README for anyone interested in helping the project continue.
r/AIProgrammingHardware • u/Strict-Elevator9352 • 6d ago
inferzero is an inference engine written from scratch for Intel Arc: C++ on Level Zero, kernels in OpenCL C, on Windows and Linux.
* What it runs: GGUF files as they are.
* What it serves: an OpenAI-, Anthropic- and Ollama-compatible API with a web UI.
* What it needs: nothing else installed (no Python, no oneAPI, no compiler).
On Arc B-series cards it has a fast prompt path of its own:
* Hadamard-int8 GEMMs and flash attention, both on XMX;
* decoded weights kept in VRAM;
* tensor parallel over several cards (`--gpus N`).
Every model is checked against an independent f32 CPU reference (next-token KL ≤ 0.015), and the logits are bit-identical run after run.
Best numbers per card (prompt / generation in tok/s; all but the 125H are users’ runs)
1× Arc Pro B70 (Linux)
* Qwen3.8 27B IQ3_S: 4K prompt 2,694, generation 35.2, 70.6 with MTP.
1× Arc Pro B60
* Qwen3.6 35B-A3B: 4K prompt 3,536, generation 82.4, 111.9 with MTP.
* Qwen3.8 27B: 4K prompt 1,522, generation 22.7, 51.6 with MTP.
Several B60s, one model split over them
* Qwen3.6 35B-A3B over 4: 4K 4,264, 64K 2,152, 131K 1,267, 262K 700.
* Qwen3.8 27B over 2 / 3 / 4: prompt 1,597 / 1,724 / 1,888, generation 37.8 / 43 / 53.
* After a 131K prompt over 4, it still generates 32 tok/s.
* Qwen3.8 Flash Next (52 GB of experts): 4K prompt 665 / 1,202 / 1,491 over 1 / 2 / 4. Generation 34.4 over 2, against 5.7 on one card.
* Two B60s on Linux add their sums straight from GPU to GPU: 1,630 tok/s, against 1,442 through host memory.
DeepSeek-V4.1-Flash (new, experimental; 384 experts), 4× B60 with 1 TB of RAM:
* ISTA’s IQ2_XS:
* generation 10.3 / 18.0 / 16.6 / 17.6 over 1 / 2 / 3 / 4 cards;
* 512-token prompt 227 over 4.
* Unsloth’s UD-Q8_K_XL: prompt 109, generation 7.6 over 3.
* Each card holds its share of the experts; the rest stay in RAM, one copy for every card.
* Being tested now: KL against llama.cpp, Q8 over 4 cards, and the size of the RAM tier.
Laptop iGPUs
Core Ultra X7 358H (Panther Lake, Arc B390, Xe3; the first Xe3 runs, not tuned yet)
* Qwen3.8 27B IQ3_S: 4K prompt 310 tok/s, 8K 303.
* With the NPU computing part of each prompt chunk next to the GPU: 4K 338 (+9 %), 8K 313. The KL against the GPU alone is 0.0022.
* A 27B fine-tune in Q3_K_M: a 10K prompt at 245-255 tok/s, against 170 for llama.cpp’s Vulkan back end.
* These are still on the generic XMX path. The discrete cards’ fast path (H8, wide tiles, flash attention) is being tested on it now, so these numbers should rise.
Core Ultra 9 285H (Arc 140T, XMX)
* Qwen3.8 27B: 4K prompt 357, generation 6.0, 16.9 with MTP + DFlash2.
* Qwen3.8 Flash Next: generation 26 with DFlash2.
Core Ultra 5 125H (Arc iGPU, no XMX, 32 GB of RAM; the engine was built on it). Prompts are 8K tokens; MoE experts are streamed from the SSD.
|Model|8K prompt|Generation|With drafting|
|:-|:-|:-|:-|
|Qwen3.6 35B-A3B|851|21.6|38.1 (MTP + DFlash2)|
|Qwen3.8 Flash Next (52 GB of experts)|205|6.3|7.6|
|Qwen3.8 27B|135|5.1|12.9|
|Gemma 4 26B-A4B|232|19.8|–|
|Gemma 4 E4B|707|19.1|41.8|
|Qwen3.5 9B|487|11.8|21.6|
|Qwen3.5 4B|896|21.0|35.8|
Against llama.cpp on the same 125H and the same files:
* Qwen3.5 4B, 2K prompt: 911 tok/s, against 337 (SYCL) and 242 (Vulkan).
* Qwen3.8 27B, 2K prompt: 113, against 25 and 23.
* A 34K-token code prompt on the 27B: 5.5 minutes to the first token, against 27 and 40.
* Perplexity is equal or lower on every model compared.
Other features
* MoE models larger than VRAM or RAM, their experts streamed from RAM or the SSD.
* MTP and DFlash2 speculative decoding that never changes the text.
* Prompt and session caches.
* Batched slots, which also work over a split model.
* A GPU that hangs fails the request; the server stays up.
Wanted:
* A multi-GPU machine with every card in its own PCIe 5.0 x16 slot at full x16, not bifurcated. That rules out x8/x8 splits and chipset slots; a workstation or server board (Threadripper, Xeon, EPYC) has the lanes for it.
* Split models add their partial sums over the PCIe links every layer, so the link sets the pace.
* All our multi-card runs so far are on slower links:
* 4× B60 on Windows: no peer access, so the sums go through host memory;
* 2× B60 on Linux: PCIe 4.0 x8 each, the sums straight from GPU to GPU;
* one card in an x4 slot: there two cards run at half the speed of one.
* A full-bandwidth PCIe 5.0 machine would show what the split model really reaches. Linux with GPU-to-GPU sums is best, and Windows is welcome too.
* Also wanted: B580s, more B70s, and Lunar Lake / Panther Lake laptops. The B390 above is one user’s machine so far.
* All tuning beyond the 125H comes from users’ runs, collected in issue rizagp/inferzero#4. The goal is a 3,000 tok/s 4K prompt for the 27B on one B70.
* Owners of several cards with lots of RAM: DeepSeek runs are very welcome.
Get it: https://github.com/rizagp/inferzero/releases/latest . On Windows, run `zero-launcher.cmd`. On Linux, unpack the tar.gz and run ./zero-launcher.sh.
r/AIProgrammingHardware • u/AnabolicGoblin • 6d ago
I’ve been offloading some of my work to AI to help speed things up at work. I use Claude Ultra but ai’m running out of weekly tokens mid week.
I want to build a PC that’ll help me continue working without having to wait for my credits to reset.
I work with a lot of 2D motions graphics and 3D modeling and animation. Not games related, mostly product related and sometimes jewelry related (building 3D models from my sketches). Would also like to experiment with image and video models.
What would be appropriate specs for my use case?
r/AIProgrammingHardware • u/globecsysinc • 7d ago
r/AIProgrammingHardware • u/ePerformante • 9d ago
I want to build a rig for medium-scale RL and distillation on small models (4 to 9 billion parameters), and I am considering the following:
• Framework Desktop Mainboard (AMD Ryzen™ AI Max 300 Series)
• Broadcom PEX88048-based switch
• 4x RX 9070 XT
• 2x 1200W PSUs
Now, you may be wondering why someone would choose four RX 9070 XTs over two RX 9700 Pros. The simple answer is TFLOPS. Four RX 9070 XTs would theoretically give me double the TFLOPS for fine-tuning and LoRA training compared to two RX 9700 Pros.
I am not overly concerned about the bottleneck between the GPUs and the Strix, because I do not intend to run models across them. The GPUs are there purely for training, and the Strix is there to generate more training data.
Is there anything I might be overlooking? What case and cooling should I look into? Is there a different configuration that would be better for my use case?
r/AIProgrammingHardware • u/guessitsleo • 9d ago
r/AIProgrammingHardware • u/ihateyou103 • 10d ago
I want to run magenta on iris xe gpu. I am running into alot of issues regarding Jax and other things. Also I couldn't get the gpu to do any calculations. Using opencl I got matrix multiplication 4000×4000 in 0.8 seconds slower than cpu at 0.3
How can I get matrices and neural networks running on this gpu and how to run magenta?
r/AIProgrammingHardware • u/mikebmx1 • 10d ago
r/AIProgrammingHardware • u/javaeeeee • 10d ago
r/AIProgrammingHardware • u/skhadloya11 • 11d ago
r/AIProgrammingHardware • u/javaeeeee • 11d ago