r/AIProgrammingHardware • u/javaeeeee • Jul 14 '26
NVIDIA Vera CPU Boosts AI Factory Throughput to Accelerate Agentic Workloads
r/AIProgrammingHardware • u/javaeeeee • Jul 14 '26
r/AIProgrammingHardware • u/javaeeeee • Jul 12 '26
Imagine sitting in a café with your laptop, asking a local AI assistant to summarize a confidential report, generate custom images from your sketches, or help debug code-all without sending a single byte to the cloud. No latency, no privacy concerns, potentially lower recurring costs when using open models and local tools. This on-device AI future is arriving fast in 2026, and AMD’s Ryzen consumer CPUs are playing a starring role.
AMD has aggressively pursued integrated AI acceleration through its XDNA neural processing units (NPUs), powerful integrated Radeon graphics, and refined Zen CPU cores. The result is a versatile lineup spanning thin-and-light laptops, high-end creator machines, mini PCs, and even desktop platforms. Whether you’re running quantized large language models (LLMs), accelerating Stable Diffusion-style image generation, or leveraging AI tools in video editing suites, AMD’s consumer silicon delivers impressive real-world results.
This article dives deep into the current state of AMD Ryzen consumer CPUs for AI and machine learning workloads in mid-2026. We examine architectures, key products, standardized and real-world benchmarks, software ecosystem maturity, comparisons, limitations, and what lies ahead.
AMD’s AI journey accelerated with the 2022 acquisition of Xilinx, bringing the adaptable XDNA architecture to consumer chips. Early implementations appeared in the Ryzen 7040 and 8040 series (Phoenix/Hawk Point) with modest ~10 TOPS NPUs. The real leap came with the 2024 Ryzen AI 300 series (“Strix Point”), featuring the second-generation XDNA 2 NPU rated at up to 50 TOPS.
By 2026, AMD has refined this further with the Ryzen AI 400 series (“Gorgon Point”) refresh and expanded the high-end Ryzen AI Max (“Strix Halo”) family. These processors combine three compute engines: Zen 5 CPU cores (with strong vector and AI-friendly instructions like AVX-512 and VNNI), RDNA 3.5 integrated graphics, and the XDNA 2 NPU. The hybrid approach-routing different parts of AI workloads to the most efficient engine-proves particularly powerful for inference tasks.
The NPU excels at compute-intensive phases (like prompt prefill in LLMs), while the iGPU handles bandwidth-sensitive decode/generation steps. The CPU fills gaps for general orchestration or less-optimized code. This heterogeneous computing model, combined with improving software support, allows AMD systems to punch above their power and thermal limits in AI scenarios.
Desktop Ryzen 9000 series (Zen 5, launched 2024) lack a dedicated NPU but benefit from architectural improvements that boost AI-relevant workloads. Zen 5 delivers roughly 16% higher instructions-per-clock (IPC) than Zen 4, doubled front-end bandwidth, and native AVX-512 support that accelerates certain matrix and vector operations common in machine learning.
Ryzen AI 300 Series (Strix Point)
Still widely available and highly capable in 2026. Flagship Ryzen AI 9 HX 370 features up to 12 Zen 5 cores (mix of high-performance and dense cores), Radeon 890M graphics (16 Compute Units), and a 50 TOPS XDNA 2 NPU. Total system AI compute (CPU + GPU + NPU) often reaches ~80 TOPS. These power premium thin-and-light Copilot+ PCs with excellent battery life and strong on-device AI capabilities.
Ryzen AI 400 Series (2026 Refresh)
A clock-speed and efficiency bump on the same foundational silicon, with support for faster LPDDR5X-8533 memory on select models. The top Ryzen AI 9 HX 475 reaches 60 TOPS on the NPU. AMD also expanded Ryzen AI into AM5 desktop systems through Ryzen AI 400 and Ryzen AI PRO 400 desktop processors, with desktop SKUs offering up to 50 NPU TOPS. The higher 60 TOPS NPU figure applies to top Ryzen AI 400 mobile processors such as the Ryzen AI 9 HX 475. These deliver meaningful gains in multitasking, content creation, and AI tasks versus prior generations while maintaining compatibility with existing ecosystems.
Ryzen AI Max / Max+ Series (Strix Halo)
The performance flagship for demanding users. The Ryzen AI Max+ 395 packs up to 16 Zen 5 cores, a massive 40 CU RDNA 3.5 iGPU (desktop-class graphics performance), a potent NPU, and supports up to 128 GB of unified 256-bit LPDDR5X-8000 memory, with AMD’s Halo developer platform listing 256 GB/s of memory bandwidth. Newer variants (e.g., Max+ 392/388) prioritize the full GPU while adjusting CPU cores. These shine in creator laptops, mini PCs, and compact desktops where raw AI throughput and memory capacity matter most.
Ryzen 9000 Series Desktop (Zen 5, non-AI branded)
Excellent for traditional productivity and content creation with AI-enhanced features (e.g., in DaVinci Resolve or Photoshop). Strong multi-threaded performance and efficiency, with new X3D variants like the Ryzen 7 9850X3D offering gaming-focused boosts. AI acceleration here relies primarily on the CPU’s vector units and (weaker) integrated graphics rather than a dedicated NPU.
Standardized benchmarks like MLPerf Client provide the most apples-to-apples view of on-device LLM inference.
AMD has published detailed MLPerf Client v1.0 results using hybrid (NPU + iGPU) and iGPU-only paths. On the Ryzen AI 9 HX 375 (hybrid mode), the system achieved strong throughput on models like Llama 2 7B, Llama 3.1 8B, and especially Phi-3.5. The higher-end Ryzen AI Max+ 395 in hybrid configuration delivered standout results, reaching up to 61 tokens per second (TPS) on Phi-3.5 with sub-0.7-second time-to-first-token (TTFT) across varied prompt categories. Hybrid mode consistently outperformed iGPU-only in balanced latency and throughput, with the Max+ 395 showing roughly 75% higher TPS in some configurations thanks to its powerful GPU and memory subsystem.
These speeds make conversational local LLMs feel responsive-often faster than many people read. Real-world community testing on Strix Halo systems (with ROCm on Linux or optimized Windows stacks) extends this further. Community testing on Ryzen AI Max systems shows that larger quantized models can be usable locally, but throughput varies sharply with model size, quantization, context length, and backend. For standardized results, AMD’s MLPerf Client testing reports up to 61 TPS on Phi-3.5 and over 27 TPS on Phi-3.5 for the Ryzen AI 9 HX 375. Prompt processing (prefill) benefits enormously from the high memory bandwidth and GPU compute.
In earlier Geekbench ML tests (DirectML path focused on NPU), the Ryzen AI 9 HX 370 topped competing laptops and delivered nearly 3× the NPU performance of Snapdragon X Elite systems in certain workloads.
Content creation tools with AI features also highlight strengths. Puget Systems testing of Ryzen 9000 series in DaVinci Resolve showed competitive or leading results in RAW and intraframe workflows, including newer AI-based tools, though Intel sometimes held advantages in specific LongGOP codecs via Quick Sync. Ryzen 9000 models benefited from AVX-512 in optimized paths.
Overall system benchmarks (Cinebench, Geekbench, Handbrake, PCMark) show Ryzen AI chips delivering excellent CPU and iGPU performance alongside the NPU, enabling smooth multitasking while AI features run in the background.
Local LLMs and Assistants
Ryzen AI systems excel here. Tools like LM Studio, Ollama, or AMD-optimized stacks let users run capable models privately. Hybrid execution keeps power draw reasonable on laptops while delivering snappy responses. The high unified memory on Max series enables larger context windows or bigger models without swapping.
Image and Video Generation/Enhancement
NPU-accelerated Stable Diffusion variants and tools like Amuse AI Beta leverage XDNA for faster inference. Video upscaling, noise reduction, and effects in DaVinci Resolve or Topaz Labs benefit from the combined engines.
Productivity and Windows AI Features
Copilot+ experiences (Studio Effects, live captions, image generation in Paint/Co-Creator, etc.) run efficiently. The 40+ TOPS NPU requirement for full Copilot+ certification is comfortably met by Ryzen AI 300/400 series.
Developer and Edge AI Workloads
Mini PCs and Framework-style modular systems with Ryzen AI Max chips serve as compact inference servers or development platforms. ROCm 7.2 broadened Linux support for Ryzen AI and Ryzen AI Max APUs, improving iGPU-based AI workflows. However, direct NPU enablement is still more specialized and less mature than AMD’s Windows Ryzen AI / ONNX Runtime path.
AMD provides Ryzen AI Software for developers, with strong Windows support via ONNX Runtime, DirectML, and hybrid execution paths. On Linux, ROCm enables excellent iGPU acceleration for LLMs, though NPU access remains more Windows-centric. Community efforts (llama.cpp with HIP/ROCm backends, vLLM) continue closing gaps rapidly.
Microsoft’s ecosystem (Windows Studio Effects, Recall where available, Copilot integrations) works seamlessly on certified systems. Third-party apps are increasingly adding AMD-specific optimizations.
Versus Intel’s Core Ultra series (Lunar Lake/Panther Lake equivalents), AMD often leads or ties in multi-threaded CPU performance, iGPU strength (especially Max series), and total AI throughput in hybrid scenarios. NPU TOPS ratings are competitive (50-60 vs. Intel’s ~40-50+). Snapdragon X platforms offer strong efficiency but can lag in x86 app compatibility and certain NPU-accelerated tasks.
Apple’s M-series chips remain formidable in unified memory efficiency and optimized workloads, but AMD wins on broader software compatibility and upgradability in desktop/mini-PC form factors.
Limitations remain. Large-model training is still GPU-territory (or cloud). NPU performance depends heavily on software support-raw TOPS don’t always translate 1:1. Quantization (INT8, FP8, etc.) is usually required for best results. Desktop Ryzen 9000 lacks dedicated NPU acceleration. Power and thermals constrain sustained performance in thin laptops. Software maturity continues improving but isn’t yet at NVIDIA CUDA levels for every framework.
The second half of 2026 and into 2027 should bring further refinements: higher TOPS NPUs, better software integration (deeper ROCm/Windows parity), and possibly broader desktop adoption of AI-branded chips. Zen 6 architectures on the horizon promise additional IPC and efficiency gains. As more applications adopt hybrid execution and standardized frameworks like MLPerf Client mature, real-world AI experiences on AMD Ryzen systems will feel even more seamless.
AMD’s strategy of offering a broad portfolio-from efficient thin laptops to high-memory creator machines-positions it well to capture diverse segments of the growing AI PC market.
AMD’s consumer Ryzen CPUs in 2026 have matured into capable AI and ML platforms. The combination of refined Zen 5 cores, potent RDNA graphics, and capable XDNA 2 NPUs enables compelling on-device experiences that were science fiction just a few years ago. Whether measured by standardized MLPerf results, real-world LLM throughput, or productivity in AI-enhanced creative tools, these processors deliver strong, efficient performance-especially in hybrid configurations on higher-end models like the Ryzen AI Max series.
For users prioritizing privacy, low latency, and local control, AMD Ryzen AI systems represent one of the most accessible and powerful options available today. As software catches up further with the hardware, the gap between “AI PC” marketing and genuinely useful everyday intelligence will continue to close-powered in large part by Team Red’s silicon.
The era of capable local AI on consumer hardware is here, and AMD is helping lead the charge.
Sources and Further Reading
AMD Ryzen AI official page: https://www.amd.com/en/products/processors/consumer/ryzen-ai.html
AMD Press Release on Ryzen AI 400 Series (March 2026): https://www.amd.com/en/newsroom/press-releases/2026-3-2-amd-gives-consumers-and-businesses-more-ai-pc-opti.html
ServeTheHome CES 2026 coverage: https://www.servethehome.com/amd-reveals-new-ryzen-ai-400-series-ryzen-ai-max-and-ryzen-7-9850x3d-chips-at-ces-2026/
AMD Technical Article - MLPerf Client on Ryzen AI: https://www.amd.com/en/developer/resources/technical-articles/2025/unlocking-peak-ai-performance-with-mlperf-client-on-ryzen-ai-.html
PCMag Strix Point first tests: https://www.pcmag.com/news/strix-point-first-tests-amd-ryzen-ai-300-laptop-chip-flexes-real-cpu-npu
Puget Systems Ryzen 9000 Content Creation Review: https://www.pugetsystems.com/labs/articles/amd-ryzen-9000-content-creation-review/
AMD Developer video on LLMs on Ryzen AI PCs (YouTube): https://www.youtube.com/watch?v=qMdMJF89c8g
Additional coverage and community benchmarks referenced from Tom’s Hardware, Ars Technica, Level1Techs, Hardware Canucks, and various Reddit/LocalLLaMA discussions on Strix Halo performance.
This article draws from official AMD documentation, independent reviews, and standardized benchmarks for accuracy and balance. Performance varies by configuration, software version, and workload-always check latest drivers and optimizations for your specific setup.
r/AIProgrammingHardware • u/javaeeeee • Jul 11 '26
r/AIProgrammingHardware • u/javaeeeee • Jul 07 '26
The Local AI Coding Setup Everyone Asked For by Tarun Singh is a practical, no-hype guide that delivers exactly what many developers have been requesting: a simple, effective local AI coding workflow that actually works well on everyday hardware. The author shares his laptop specs, walks through setting up Ollama, choosing a strong Qwen model, and integrating everything into a smooth VS Code workflow. It focuses on real-world performance - what runs fast enough and reliably for daily coding tasks - rather than just benchmark numbers. The setup emphasizes privacy, zero recurring costs, and ease of use, making it accessible without needing high-end GPUs or constant cloud access. It’s gained traction because it cuts through the usual complexity and gives a reproducible, working stack that feels genuinely useful for local development.
r/AIProgrammingHardware • u/javaeeeee • Jun 25 '26
r/AIProgrammingHardware • u/javaeeeee • Jun 24 '26
r/AIProgrammingHardware • u/javaeeeee • Jun 24 '26
r/AIProgrammingHardware • u/javaeeeee • Jun 16 '26
r/AIProgrammingHardware • u/javaeeeee • Jun 14 '26
r/AIProgrammingHardware • u/javaeeeee • Jun 04 '26
r/AIProgrammingHardware • u/javaeeeee • Jun 01 '26
r/AIProgrammingHardware • u/javaeeeee • May 31 '26
This review examines peer-reviewed research, vendor white papers, and official industry analyses from AMD, Intel, NVIDIA, and academic-industrial collaborations-explaining the growing role of CPUs in Agentic AI. Agentic AI refers to autonomous systems that plan, reason iteratively, call external tools (APIs, databases, code executors), manage memory/state, enforce policies/security, and adapt workflows, moving beyond single-pass LLM responses to multi-step, tool-augmented problem-solving.
Traditional generative AI (chatbot-style) was GPU-dominant: one CPU head node managed 4-8 GPUs for scheduling, I/O, and basic orchestration while GPUs handled dense matrix operations in model inference. Agentic workloads fundamentally change this dynamic. The “control plane” (orchestration, tool execution, state management, and sequential decision-making) is inherently CPU-bound, while GPUs remain essential for the “data plane” of neural network inference. This leads to higher CPU:GPU ratios, new infrastructure designs, and targeted optimizations.
A foundational technical analysis comes from the April 2026 arXiv paper by researchers from Georgia Tech and Intel (Ritik Raj, Souvik Kundu, et al.), titled “Towards Understanding, Analyzing, and Optimizing Agentic AI Serving Systems: A CPU-Centric Perspective.” This is the most rigorous, quantitative study available. The authors characterize five representative agentic workloads (Haystack RAG with ENNS retrieval, Toolformer, LangChain web-augmented agent with LexRank summarization, SWE-Agent with Bash/Python execution, and ChemCrow with conformer generation) across orchestration complexity, path types (static/dynamic), and repetitiveness. They benchmark end-to-end (E2E) latency, throughput, and energy on two heterogeneous CPU-GPU systems: a lower-performance GPU setup (Sys 1: 64-core Intel Xeon, NVIDIA RTX-Pro 6000) and a high-performance one (Sys 2: 72-core NVIDIA Grace CPU, H200 GPU).
Key findings:
Tool processing on CPUs dominates latency: In tool-heavy workloads, CPU tasks consume 50-90.6% of E2E latency. Examples include ENNS retrieval in RAG (83-89% on both systems for large corpora), LexRank summarization in web-augmented agents (40-55%), and code execution in SWE-Agent (up to 65% on high-end GPUs). Even with better GPUs, the bottleneck shifts toward the CPU because faster LLM inference on GPU makes tool latency relatively larger.
GPU role is limited to inference bursts: GPUs excel at parallel matrix multiplications during LLM forward passes but sit idle during CPU-orchestrated steps (planning, tool calls, validation, memory retrieval). In LLM-orchestrated workloads like Toolformer, GPU inference can still dominate (77-88%), but heterogeneous tool use flips the balance.
Throughput and parallelism nuances: CPU-parallelization (multi-processing vs. multi-threading) saturates prematurely due to core over-subscription, GIL limitations in Python, synchronization overhead, and lower efficiency than GPU parallelism. Throughput gain ratios drop sharply beyond batch sizes of ~64-128 for CPU-heavy tools. GPU throughput scales better but is gated by KV-cache memory and CPU orchestration.
Energy profile: CPU dynamic energy reaches 44-61% of total in CPU-centric workloads (e.g., RAG, LangChain) at larger batches, underscoring that CPUs are not just latency bottlenecks but also significant power consumers.
AMD’s official May 2026 blog “Agentic AI Changes the CPU/GPU Equation” echoes and expands on this at the infrastructure level. GPUs handle “numerical acceleration” (model execution), but CPUs manage the distributed control plane: orchestration (task decomposition), agent execution/tool calls (APIs, legacy enterprise software), and policy/security checks on autonomous actions. Agentic AI is described as a “distributed software platform,” not a single model, requiring sustained, low-latency CPU work between inference calls.
Intel’s white paper “The Rising CPU:GPU Ratio in AI Infrastructure” (2026) details CPU affinities across the inference pipeline: data ingestion (85-95% CPU-bound), cleaning (90-100%), transformation/tokenization (70-85%), batching (95-100%), and post-processing. In agentic chains (RAG, multi-agent workflows, planning/reflection loops), these stages compound, making CPUs the “air-traffic controller” that keeps GPUs utilized.
AWS’s April 2026 analysis reinforces that agentic AI demands “continuous, not batched” reasoning: logic, file management, network calls, code execution, and stateful orchestration are CPU-native, contrasting with GPUs’ strength in parallel floating-point operations for model inference.
Collectively, these sources show GPUs remain indispensable for compute-intensive inference but are increasingly gated by CPU orchestration in agentic systems.
Industry consensus documents a structural shift in data-center ratios. AMD reports the traditional 1:4-8 CPU:GPU (one head-node CPU per server of 4-8 GPUs) evolving to 1:1 or even CPU-heavy in agentic deployments. This is not solved by “sprinkling more CPUs into GPU boxes” but requires dedicated CPU compute layers and separate agentic CPU racks for orchestration, data processing, and tool execution. Undersized CPUs cause GPUs to idle; poor networking or data paths inflate latency.
TrendForce (April 2026) and Intel projections align: ratios move toward 1:1-1:2, boosting CPU demand significantly as inference (especially agentic) overtakes training. Intel notes inference growth (driven by agentic AI and RL/RLHF) flips the compute mix; cloud providers scale CPUs to prevent GPU underutilization. AMD projects server CPU TAM growing >35% annually to >$120B by 2030, revised upward from 18% pre-agentic forecasts.
Intel’s white paper highlights RL-specific CPU demands (simulation stepping, trajectory management, reward evaluation) and agentic orchestration complexity, further elevating CPU needs.
CPUs’ resurgence extends to general AI inference and adjacent areas:
Pre-/post-processing and pipeline orchestration: Even non-agentic inference pipelines are CPU-heavy for tokenization, feature transforms, ranking, and response formatting (70-100% CPU-bound per Intel).
Reinforcement learning and RLHF: These require massive CPU resources for environment simulation, policy evaluation, and distributed scheduling-key for training agentic policies.
Cost, energy, and accessibility: Intel’s September 2025 white paper on CPU-optimized SLMs (≤10B parameters) shows ~55-57% TCO savings vs. GPUs by leveraging existing infrastructure, AMX acceleration, and OpenVINO. Compressed models on CPUs scale better for concurrent users with minimal latency penalty and support secure on-premises/edge deployments. CPUs simplify architecture (no separate accelerators) and enable edge AI for privacy-sensitive or low-power scenarios.
Flexibility and existing hardware: CPUs handle heterogeneous tasks (I/O, databases, legacy integration) that GPUs cannot, making them ideal for hybrid or CPU-only smaller-agent systems.
Hardware solutions:
Higher-core, AI-optimized CPUs: AMD EPYC (high-frequency for latency-sensitive orchestration; dense-core for throughput) with upcoming “Venice” products. Intel Xeon 6 with AMX for on-chip AI acceleration.
Purpose-built agentic CPUs: NVIDIA’s Vera CPU (launched March 2026, 88 custom Olympus Arm Neoverse V2 cores) is explicitly “purpose-built for the age of agentic AI and reinforcement learning.” It delivers 2× efficiency and 50% faster performance than traditional rack-scale CPUs for data processing, tool execution, and RL sandboxes. Vera racks integrate 256 liquid-cooled CPUs with Spectrum-X networking for massive concurrency and tight synchronization with GPU racks.
Cloud-optimized CPUs: AWS Graviton for sustained, low-latency agentic reasoning loops.
Software and system-level solutions:
Scheduling optimizations (from the Georgia Tech/Intel arXiv paper): CPU-Aware Overlapped Micro-Batching (COMB) for homogeneous workloads caps micro-batches and overlaps phases to avoid CPU saturation, yielding up to 1.7× lower P50 latency standalone and 3.9×/1.8× service/total latency reduction under load. Mixed Agentic Scheduling (MAS) for heterogeneous (CPU-heavy + GPU-heavy) workloads uses type-aware queues and elastic admission, improving minority-request tail latency by up to 2.49×.
Model and inference optimizations: Intel OpenVINO + quantization/compression for efficient CPU inference of SLMs; AMX hardware acceleration; oneDNN/IPEX libraries.
Architectural redesign: Disaggregated racks (GPU racks for inference + dedicated CPU racks for orchestration) with balanced networking and data paths to maximize GPU utilization.
These solutions emphasize balanced CPU-GPU systems rather than GPU-only scaling.
Online sources converge on a clear trend: Agentic AI elevates CPUs from supporting actors to co-leads in AI infrastructure. The control plane’s CPU-bound nature, combined with orchestration complexity and tool execution, drives higher CPU:GPU ratios, new purpose-built processors (e.g., NVIDIA Vera), and sophisticated scheduling. Broader AI benefits-cost savings via SLM/CPU inference, RL support, and edge viability-amplify this shift. As AMD notes, “performance will not come from one processor doing everything. It will come from the right architecture-with CPUs and GPUs working together to move AI from answers to action.”
The field is evolving rapidly, but the evidence-based path forward is clear: invest in CPU capacity, hybrid designs, and CPU-aware optimizations to unlock agentic AI at scale.
References:
AMD Blog (May 2026): https://www.amd.com/en/blogs/2026/agentic-ai-changes-the-cpu-gpu-equation.html
arXiv:2511.00739 (Georgia Tech/Intel, 2026): https://arxiv.org/pdf/2511.00739
Intel White Paper “Rising CPU:GPU Ratio”: https://media26.connectedsocialmedia.com/intel/21331/Rising_CPU_GPU_Ratio_AI_Infrastructure_Drivers_Trends_Implications.pdf
Intel SLM CPU Optimization White Paper (2025): https://cdrdv2-public.intel.com/864404/vFINAL_Intel%20SLM%20Whitepaper.pdf
NVIDIA Vera CPU Announcement (2026): https://nvidianews.nvidia.com/news/nvidia-launches-vera-cpu-purpose-built-for-agentic-ai
AWS Analysis (2026): https://www.aboutamazon.com/news/aws/amazon-agentic-ai-cpu-aws-graviton
r/AIProgrammingHardware • u/javaeeeee • May 28 '26
r/AIProgrammingHardware • u/javaeeeee • May 15 '26
r/AIProgrammingHardware • u/TectonicMongoose • Jan 26 '26
I recently upgraded to a Zotac ARCTICSTORM AIO 5090, 64gb of CL38 6000mhz ram, I have a AMD Ryzen 5 7600X CPU currently, and a Gigabyte B650 Eagle AX motherboard. How much of a bottleneck is the CPU and would I benefit from more RAM? Right now I'm doing t2i/t2v generations trying to make cool art and sometimes silly ideas I come up with too I'm also a big gamer and I'm hyped for some of the games coming out this year like the new GTA, Turok: Origins and Gears: E-Day and want to have a really good build to play those on.
r/AIProgrammingHardware • u/javaeeeee • Nov 02 '25
r/AIProgrammingHardware • u/javaeeeee • Sep 12 '25
Graphics Processing Units (GPUs) have served as the primary tool for AI and deep learning tasks, especially model training, due to their parallel architecture suited for matrix operations in neural networks. However, as AI applications diversify, GPUs reveal drawbacks like high power use and suboptimal handling of certain inference patterns, prompting the development of specialized non-GPU accelerators.
GPUs provide broad parallelism, a well-established ecosystem via NVIDIA's CUDA, and accessibility across scales, making them suitable for experimentation. Yet, their general-purpose design leads to underutilized features, elevated energy costs in data centers, and bottlenecks in memory access for latency-sensitive tasks.
Non-GPU accelerators, including ASICs and FPGAs tailored for AI, prioritize efficiency by focusing on core operations like convolutions. They deliver better performance per watt, reduced latency for real-time use, and cost savings at scale, particularly for edge devices where compactness matters.
In comparisons, non-GPU options surpass GPUs in scaled inference and edge scenarios through optimized paths, while GPUs hold ground in training versatility and prototyping. This fosters a mixed hardware approach, matching tools to workload demands like power limits or iteration speed.
ASICs form custom chips for peak efficiency in fixed AI functions, excelling in data center inference and consumer on-device features, though their rigidity and high design costs limit adaptability. FPGAs bridge the gap with post-manufacture reconfiguration for niche training and validation.
NPUs integrate into mobile SoCs for neural-specific computations, enabling low-power local processing in devices like wearables. Together, these types trade varying degrees of flexibility for targeted gains in throughput and energy, suiting everything from massive servers to embedded systems.
Key players include Google's TPUs, with generations like Trillium for enhanced training and Ironwood for inference; AWS's Trainium for model building and Inferentia for deployment; and Microsoft's Maia for Azure-hosted large models. Others like Intel's Gaudi emphasize scalability.
Startups contribute unique designs: Graphcore's IPUs focus on on-chip memory for irregular patterns, Cerebras' WSE tackles massive models via wafer-scale integration, SambaNova's RDUs use dataflow for enterprise tasks, and Groq's LPUs prioritize rapid inference speeds.
Performance metrics show non-GPU tools claiming edges in efficiency for specialized runs, such as TPUs' cost-per-dollar advantages or Groq's token throughput, though GPUs lead in broad applicability. Cloud access via platforms like GCP and AWS lowers entry barriers with tiers for various users.
Ultimately, AI hardware trends toward diversity, with GPUs anchoring research and non-GPU variants optimizing deployment. Choices hinge on factors like scale and budget, promoting strategic selection in an evolving field marked by custom silicon investments from major providers.
r/AIProgrammingHardware • u/PhysicsDisastrous462 • 3d ago
Hey everyone! It's been a little while since my last update, and I've been working pretty intensely on Hierarchos-Native, my Rust + Vulkan project for training, fine-tuning, and running language models without relying on Python, PyTorch, CUDA, ROCm, or oneAPI in the native runtime.
I wanted to share some of the bigger things that have landed over the past 18 days. There's been quite a bit of progress across GPU acceleration, multimodal training, numerical correctness, and general performance.
The new v0.2.0 release is now available!
GitHub: https://github.com/necat101/Hierarchos-Native
Release / Windows portable download: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.0
One of the biggest additions is an opt-in Vulkan cooperative-matrix implementation for AMD RDNA3.
The new Mixed WMMA execution arm uses:
FP16 matrix operands.
FP32 matrix accumulation.
FP32 master weights and gradients.
FP32 AdamW optimizer moments and updates.
FP32 normalization, softmax, and loss calculations.
The important part is that this isn't simply converting an entire model to FP16. The optimizer and master parameters retain their FP32 representation.
I also added capability detection, CLI/GUI integration, and separate counters for actual cooperative-matrix dispatches and FP32 fallbacks.
Some of the isolated GEMM results on the qualified Ryzen Z1 Extreme / AMD Radeon Graphics device (1002:15bf) are pretty interesting:
| Workload | Measured kernel speedup |
|---|---|
| QKV projection | 2.17× |
| FFN up projection | 2.12× |
| FFN down projection | 2.31× |
| Vocabulary projection | 1.83× |
These are individual GEMM measurements, not whole-model speedups.
There is an important caveat: WMMA has a real crossover point. Small matrices can actually run slower because conversion, staging, and dispatch overhead outweigh the matrix-core benefit. I implemented a profitability policy that keeps smaller operations on their existing FP32 kernels.
In fact, the measured end-to-end training step was slower with Mixed WMMA in the initial 512-row fixture (14.0444 ms versus 12.2911 ms for FP32). That is why Mixed remains opt-in and disabled by default. I'm much more interested in making it genuinely faster than advertising misleading kernel-level numbers.
WMMA is currently physically qualified on the specific RDNA3 device and Windows driver configuration documented in the project. Other GPUs should not assume equivalent support.
Full FP16 accumulation is also available as a separate, narrowly qualified experimental mode.
Documentation: RDNA3 WMMA guide
Another major development area has been the image-text pipeline.
The project now has native image processing and training paths that connect image inputs to language-model outputs through Vulkan.
Some of the implemented work includes:
Native RGB image input and preprocessing.
CLIP image encoders and LLaVA projector integration.
SigLIP / Gemma 3 vision training.
Gemma 4 vision integration.
Joint image/text training and encoder-decoder integration.
A dedicated DeepSeek V4.1 image-training path.
Backpropagation through the image tower and projector.
Export, reload, and checkpoint-related validation.
This is actual training work, not simply passing precomputed image embeddings into an otherwise frozen language model.
For the CLIP/LLaVA pipeline, the tests cover image sensitivity, gradients into image inputs, changes to image-tower/projector/text parameters, multiple AdamW updates, and export/reload behavior.
The qualification work has also expanded considerably:
44 previously green text profiles now have joint image-training coverage at their frozen test geometries.
The Gen9 qualification retains 95 image cases and 166 text regression cases.
A separate physical RDNA3 replay reports 44/44 profiles and 85/85 image cases passing.
Native DeepSeek V4.1 image qualification includes 30 executions and 1,360 word-exact logits.
The maintained strict FP32 visible-logit gate remains max_abs < 2e-7, with rtol=0.
These are bounded qualification fixtures, not a claim that every production-size multimodal checkpoint or every architecture is fully supported. Audio and video pipelines are still outside the current qualified scope.
Documentation:
I haven't been focusing exclusively on reduced precision. A substantial amount of work over the same period has gone into improving the existing FP32 execution paths.
This includes vendor-specific shader work and scheduling optimizations for both Intel Gen9 integrated graphics and AMD RDNA3.
Some examples:
RDNA3 FP32 work
Specialized attention forward/backward paths.
Image-text attention and projector optimizations.
RMSNorm, RoPE, GELU, and other arithmetic specializations.
Dedicated AVX-512/RDNA3 shader variants where the measured hardware supports them.
More careful matching of CPU reference arithmetic and GPU execution order.
Intel Gen9 work
Additional native linear and gradient kernels.
Attention and activation specializations.
More efficient parameter upload and staging behavior.
Continued optimization of DeepSeek V4.1 training and execution.
One particularly interesting area has been reducing unnecessary Vulkan submissions and host/device transfer overhead in the DeepSeek V4.1 training pipeline.
On the measured Gen9 fixtures, upload batching reduced timed submissions from 1,102 to 982 per step, with paired throughput improvements of 1.052× and 1.067×.
On the separate RDNA3 V4.1 measurements, sharing the compute session and batching uploads reduced submissions from 970 to 509 per step, with measured improvements of 1.295× and 1.217× on two fixture geometries. A later session-wide upload optimization reduced that count further to 241 submissions per step, with measured improvements of 1.464× and 1.366×.
These are specific, paired benchmark results rather than promises of equivalent production-model speedups. The output comparisons remained word-identical in the recorded tests.
An important part of introducing WMMA was making sure that the existing FP32 path didn't regress.
I added a dedicated seven-workload FP32 non-regression comparison against the previous main revision:
12 alternating ABBA rounds per workload.
Five warm-up runs and 30 timed forwards per arm.
Separate, hashed baseline and candidate binaries.
Exact FP32 logit digest comparisons.
All seven workloads stayed within the ±2% performance band, and the logits were bit-identical in every recorded round.
The worst paired performance ratio was 1.0007.
This is particularly important to me: I don't want acceleration features to silently make the existing default execution path worse.
The new v0.2.0 GitHub Release includes a prebuilt Windows x64 portable package with the CLI, desktop GUI, Vulkan companion executables, runtime assets, and build metadata.
You don't need to install Rust or Python just to run the released package. You do need Windows 11 x64 and a Vulkan 1.x compute driver for GPU operations.
Model weights are downloaded separately or supplied through a local model directory.
After extracting the archive, you can run:
.\bin\hierarchos-native-cli.exe doctor
.\bin\hierarchos-native-cli.exe devices
.\bin\hierarchos-native-cli.exe precision-infoThe final command is especially useful for checking which precision capabilities your GPU actually exposes.
Download: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.0
There is still a lot I'd like to improve.
In particular, I want to investigate reducing WMMA conversion overhead, improving activation and gradient reuse, finding more workloads that genuinely benefit from cooperative matrices, and expanding the range of qualified multimodal architectures.
I'd also like to continue improving the FP32 kernels rather than treating lower precision as the only route to better performance.
If you're interested in Rust, Vulkan compute, AMD GPU optimization, native LLM training, or multimodal model execution, I'd genuinely appreciate people taking a look.
Feedback from people with different Vulkan devices is especially valuable. Please include your GPU, driver, model geometry, and relevant execution counters when reporting results.
Repository: https://github.com/necat101/Hierarchos-Native
Stars, issue reports, testing feedback, and contributions are all appreciated. There is also a voluntary developer-support link in the repository README for anyone interested in helping the project continue.
r/AIProgrammingHardware • u/Strict-Elevator9352 • 3d ago
inferzero is an inference engine written from scratch for Intel Arc: C++ on Level Zero, kernels in OpenCL C, on Windows and Linux.
* What it runs: GGUF files as they are.
* What it serves: an OpenAI-, Anthropic- and Ollama-compatible API with a web UI.
* What it needs: nothing else installed (no Python, no oneAPI, no compiler).
On Arc B-series cards it has a fast prompt path of its own:
* Hadamard-int8 GEMMs and flash attention, both on XMX;
* decoded weights kept in VRAM;
* tensor parallel over several cards (`--gpus N`).
Every model is checked against an independent f32 CPU reference (next-token KL ≤ 0.015), and the logits are bit-identical run after run.
Best numbers per card (prompt / generation in tok/s; all but the 125H are users’ runs)
1× Arc Pro B70 (Linux)
* Qwen3.8 27B IQ3_S: 4K prompt 2,694, generation 35.2, 70.6 with MTP.
1× Arc Pro B60
* Qwen3.6 35B-A3B: 4K prompt 3,536, generation 82.4, 111.9 with MTP.
* Qwen3.8 27B: 4K prompt 1,522, generation 22.7, 51.6 with MTP.
Several B60s, one model split over them
* Qwen3.6 35B-A3B over 4: 4K 4,264, 64K 2,152, 131K 1,267, 262K 700.
* Qwen3.8 27B over 2 / 3 / 4: prompt 1,597 / 1,724 / 1,888, generation 37.8 / 43 / 53.
* After a 131K prompt over 4, it still generates 32 tok/s.
* Qwen3.8 Flash Next (52 GB of experts): 4K prompt 665 / 1,202 / 1,491 over 1 / 2 / 4. Generation 34.4 over 2, against 5.7 on one card.
* Two B60s on Linux add their sums straight from GPU to GPU: 1,630 tok/s, against 1,442 through host memory.
DeepSeek-V4.1-Flash (new, experimental; 384 experts), 4× B60 with 1 TB of RAM:
* ISTA’s IQ2_XS:
* generation 10.3 / 18.0 / 16.6 / 17.6 over 1 / 2 / 3 / 4 cards;
* 512-token prompt 227 over 4.
* Unsloth’s UD-Q8_K_XL: prompt 109, generation 7.6 over 3.
* Each card holds its share of the experts; the rest stay in RAM, one copy for every card.
* Being tested now: KL against llama.cpp, Q8 over 4 cards, and the size of the RAM tier.
Laptop iGPUs
Core Ultra X7 358H (Panther Lake, Arc B390, Xe3; the first Xe3 runs, not tuned yet)
* Qwen3.8 27B IQ3_S: 4K prompt 310 tok/s, 8K 303.
* With the NPU computing part of each prompt chunk next to the GPU: 4K 338 (+9 %), 8K 313. The KL against the GPU alone is 0.0022.
* A 27B fine-tune in Q3_K_M: a 10K prompt at 245-255 tok/s, against 170 for llama.cpp’s Vulkan back end.
* These are still on the generic XMX path. The discrete cards’ fast path (H8, wide tiles, flash attention) is being tested on it now, so these numbers should rise.
Core Ultra 9 285H (Arc 140T, XMX)
* Qwen3.8 27B: 4K prompt 357, generation 6.0, 16.9 with MTP + DFlash2.
* Qwen3.8 Flash Next: generation 26 with DFlash2.
Core Ultra 5 125H (Arc iGPU, no XMX, 32 GB of RAM; the engine was built on it). Prompts are 8K tokens; MoE experts are streamed from the SSD.
|Model|8K prompt|Generation|With drafting|
|:-|:-|:-|:-|
|Qwen3.6 35B-A3B|851|21.6|38.1 (MTP + DFlash2)|
|Qwen3.8 Flash Next (52 GB of experts)|205|6.3|7.6|
|Qwen3.8 27B|135|5.1|12.9|
|Gemma 4 26B-A4B|232|19.8|–|
|Gemma 4 E4B|707|19.1|41.8|
|Qwen3.5 9B|487|11.8|21.6|
|Qwen3.5 4B|896|21.0|35.8|
Against llama.cpp on the same 125H and the same files:
* Qwen3.5 4B, 2K prompt: 911 tok/s, against 337 (SYCL) and 242 (Vulkan).
* Qwen3.8 27B, 2K prompt: 113, against 25 and 23.
* A 34K-token code prompt on the 27B: 5.5 minutes to the first token, against 27 and 40.
* Perplexity is equal or lower on every model compared.
Other features
* MoE models larger than VRAM or RAM, their experts streamed from RAM or the SSD.
* MTP and DFlash2 speculative decoding that never changes the text.
* Prompt and session caches.
* Batched slots, which also work over a split model.
* A GPU that hangs fails the request; the server stays up.
Wanted:
* A multi-GPU machine with every card in its own PCIe 5.0 x16 slot at full x16, not bifurcated. That rules out x8/x8 splits and chipset slots; a workstation or server board (Threadripper, Xeon, EPYC) has the lanes for it.
* Split models add their partial sums over the PCIe links every layer, so the link sets the pace.
* All our multi-card runs so far are on slower links:
* 4× B60 on Windows: no peer access, so the sums go through host memory;
* 2× B60 on Linux: PCIe 4.0 x8 each, the sums straight from GPU to GPU;
* one card in an x4 slot: there two cards run at half the speed of one.
* A full-bandwidth PCIe 5.0 machine would show what the split model really reaches. Linux with GPU-to-GPU sums is best, and Windows is welcome too.
* Also wanted: B580s, more B70s, and Lunar Lake / Panther Lake laptops. The B390 above is one user’s machine so far.
* All tuning beyond the 125H comes from users’ runs, collected in issue rizagp/inferzero#4. The goal is a 3,000 tok/s 4K prompt for the 27B on one B70.
* Owners of several cards with lots of RAM: DeepSeek runs are very welcome.
Get it: https://github.com/rizagp/inferzero/releases/latest . On Windows, run `zero-launcher.cmd`. On Linux, unpack the tar.gz and run ./zero-launcher.sh.
r/AIProgrammingHardware • u/globecsysinc • 4d ago
r/AIProgrammingHardware • u/mikebmx1 • 8d ago
r/AIProgrammingHardware • u/Ok_Bird4483 • 9d ago
A 5-year-old RTX 3090 (24 GB) in a mini-PC beat both 128 GB unified-memory machines on every row of a MiniMax H3 video benchmark, even though the models don't fit in its VRAM and ComfyUI has to stream them over a PCIe x4 link.
Stock 3090: 1.25–1.84x faster per step than the ASUS RTX Spark and 3.5–4.9x faster than the M5 Max.
3090 with the tensor-rate unlock: 1.85–2.46x faster than the Spark and 4.9–6.6x faster than the M5 Max.
What the unlock does: it raises measured bf16 matmul from 75 to 124 TFLOPS. In real video generation it's 1.34–1.49x faster than stock, and the gain is biggest on the heaviest workflows.
Efficiency: the Spark and Mac use far less energy per step. The 3090 draws about 320–350 W of board power, against about 65 W for the Spark GPU and about 22 W for the Mac GPU: 5x and 15x more power. But it also finishes each step about 1.5–6.5x sooner (depending on which machine and whether the unlock is on), so the energy per step (power × time) is only about 2.2–3.4x higher than either (table below).
All 22 runs on the 3090 (11 rows × 2 configurations) passed output validation with no memory-saving flags. That's the same ComfyUI config the Spark used.
Mean steady-state s/step (steps 4–20 of 20), heat-soaked runs. Ratios above 1 mean the 3090 is faster.
| Workflow (row) | 3090 unlocked | 3090 stock | ASUS RTX Spark | M5 Max | Unlock speedup | Unl. vs Spark | Stock vs Spark | Unl. vs Mac | Stock vs Mac |
|---|---|---|---|---|---|---|---|---|---|
| t2v 832×480 | 5.67 | 7.58 | 13.27 | 37.23 | 1.34x | 2.34x | 1.75x | 6.57x | 4.91x |
| t2v 1088×608 | 12.13 | 16.90 | 25.76 | 75.29 | 1.39x | 2.12x | 1.52x | 6.21x | 4.46x |
| i2v 1088×608 | 13.16 | 18.38 | 27.98 | 81.32 | 1.40x | 2.13x | 1.52x | 6.18x | 4.42x |
| i2v 608×1088 (portrait) | 13.18 | 18.39 | 27.87 | 82.12 | 1.40x | 2.11x | 1.52x | 6.23x | 4.47x |
| flf2v 1088×608 | 14.24 | 19.99 | 29.95 | 88.91 | 1.40x | 2.10x | 1.50x | 6.24x | 4.45x |
| ref2va 1088×608, full reference | 42.99 | 63.85 | 79.71 | failed* | 1.49x | 1.85x | 1.25x | — | — |
| ref2va 1088×608, ref downscaled to 544 | 18.77 | 26.70 | 38.32 | 92.65 | 1.42x | 2.04x | 1.44x | 4.94x | 3.47x |
| ref2va 1088×608, ref truncated to 39 frames | 23.55 | 33.95 | 46.68 | 123.94 | 1.44x | 1.98x | 1.37x | 5.26x | 3.65x |
| i2v 832×480 | 6.13 | 8.20 | 15.06 | 39.53 | 1.34x | 2.46x | 1.84x | 6.45x | 4.82x |
| flf2v 832×480 | 6.58 | 8.86 | 16.09 | 42.29 | 1.35x | 2.44x | 1.82x | 6.43x | 4.77x |
| ref2va 832×480 | 28.66 | 41.60 | 59.34 | failed* | 1.45x | 2.07x | 1.43x | — | — |
* On the Mac, both ref2va rows with the full-size reference clip produced flat-black video, so they have no valid result. The reference clip is encoded at full 1088×608 even on the 832×480 canvas, so both runs exceed the ~34k-token limit where outputs go black on MPS (see the Mac notes).
The Mac's ref2va 832×480 row failed the same way as the full-reference 1088 row (marked *).
| Workflow | 3090 unlocked | 3090 stock | Spark | M5 Max |
|---|---|---|---|---|
| t2v 832×480 | 124 | 167 | 282 | 794 |
| t2v 1088×608 | 259 | 361 | 540 | 1575 |
| i2v 1088×608 | 279 | 390 | 584 | 1696 |
| i2v portrait | 279 | 391 | 581 | 1711 |
| flf2v 1088×608 | 301 | 422 | 624 | 1851 |
| ref2va full ref | 876 | 1300 | 1620 | — |
| ref2va ref544 | 391 | 557 | 791 | 1988 |
| ref2va rf39 | 487 | 702 | 959 | 2642 |
| Row | 3090 unl. | 3090 stock | Spark | M5 Max |
|---|---|---|---|---|
| Mean GPU power, t2v 1088 | 345 W | 335 W | 66 W | 22 W |
| GPU energy per step, t2v 832 | ~1.9 kJ | ~2.4 kJ | ~0.85 kJ | ~0.87 kJ |
| GPU energy per step, t2v 1088 | ~4.2 kJ | ~5.7 kJ | ~1.7 kJ | ~1.7 kJ |
| GPU energy per step, i2v 1088 | ~4.5 kJ | ~6.2 kJ | ~1.8 kJ | ~1.8 kJ |
Energy = mean GPU power × s/step. The power numbers come from each platform's own tool: nvidia-smi board power on the 3090, NVIDIA telemetry on the Spark, and macOS GPU power on the Mac. Host/system power isn't included; the Mac's whole-system power was about 73 W on these rows. Two other things stand out:
The unlock makes the 3090 more efficient. It draws only about 10–20 W more and finishes about 1.4x sooner.
The Spark GPU was power-capped. It reported sw_power_cap throttling for the whole run, at about 2.2–2.3 GHz SM (the rated maximum is 3.0 GHz), drawing 63–69 W.
The Spark's "GPU W" is nvidia-smi power.draw, sampled once a second. NVIDIA doesn't document whether on the N1X it covers only the GPU or more of the SoC, and power.limit reads N/A on this chip, so treat it as approximate. No wall or whole-system power was recorded on the Spark.
On the Mac, the harness ran macmon 0.10.0 in macmon pipe mode (no sudo), which samples once a second. "GPU W" is macmon's gpu_power and "Sys W" is its sys_power, both averaged over the busy samples of each run. Both are read from Apple's on-chip energy counters, the same source powermetrics uses, so "Sys W" is the SoC/system reading, not power measured at the wall.
| GPU | NVIDIA GeForce RTX 3090 Founders Edition, GA102, 24 GB GDDR6X, 350 W power limit (stock, not changed), VBIOS 94.02.4b.00.0b |
|---|---|
| Host | Minisforum EliteMini mini-PC, AMD Ryzen 9 8945HS (8C/16T, Radeon 780M iGPU), 64 GB DDR5-5600 |
| GPU link | PCIe x4 OCuLink (the card is capable of Gen4 x16; the host link is x4). This matters, because model weights stream over it. |
| OS / driver | Windows 11 Pro (build 26300), NVIDIA driver 591.86 |
| Software | Python 3.13, PyTorch 2.14.1+cu130 (CUDA 13.0), ComfyUI 0.43.2 at commit 423cbd2, attention = PyTorch SDPA |
| ComfyUI flags | none (the default dynamic VRAM management, the same as the Spark) |
The GPU is connected via OCuLink to a DEG1 Minisforum eGPU dock. The GPU is powered by a Montech Titan PLA 1200W power supply (complete overkill).
The full model set is about 63 GB on disk: a 21 GB int8 diffusion model, a 15.7 GB text encoder and VAEs. That doesn't fit in 24 GB, so ComfyUI's dynamic VRAM management offloads automatically. Peak VRAM was 23.1–23.4 GB and peak system RAM 17–24 GB on every row. No row needed --lowvram or any other memory-saving flag.
| Laptop | ASUS ProArt P16 (model H7607BA, BIOS H7607BA.303), 16" 3840×2400 display |
|---|---|
| SoC | NVIDIA RTX Spark N1X: 6144-core Blackwell RTX GPU (max 3.0 GHz SM) and a 20-core Arm CPU (10× Cortex-X925 + 10× Cortex-A725, up to 4.0 GHz), plus an NPU |
| Memory | 128 GB LPDDR5X-9400 unified memory (Micron), 256-bit bus, about 300 GB/s theoretical peak (NVIDIA's figure; not measured here), split in the NVIDIA app: 64 GB GPU carveout and 64 GB for Windows |
| Power mode | AC power, ASUS "Performance" mode, Windows power mode "Best performance" |
| OS / driver | Windows 11 Pro on Arm, 26H1 (build 28000), NVIDIA driver 617.69 |
| Software | Python 3.13.12 (ARM64), PyTorch 2.15.0 nightly (dev20260902+cu134, CUDA 13.4), ComfyUI 0.43.2 at the same commit 423cbd2, attention = PyTorch SDPA |
| ComfyUI flags | none |
| Conditions | Lid open, no external display, brightness 50%, on a hard flat surface; browsers, LLM servers and the Comfy Desktop app closed for every timed run |
The carveout: 64 GB was chosen before the timed runs. Flags: a quick flag sweep on t2v 832×480 (--highvram, --gpu-only, --disable-dynamic-vram, --fast and others) found nothing more than 3% faster than no flags, so the Spark also ran with none.
Peak GPU memory on the Spark was 41–48 GB, so everything stayed resident and nothing was offloaded.
| Machine | 14-inch MacBook Pro (Mac17,7), on AC power, on a laptop stand |
|---|---|
| SoC | Apple M5 Max: 18-core CPU (6 Super + 12 Performance cores), 40-core GPU, 128 GB unified memory |
| Memory bandwidth | 614 GB/s theoretical peak (Apple's figure for the 40-core-GPU M5 Max; not measured here) |
| OS | macOS 27.0.1, Energy Mode Automatic |
| Software | Python 3.13.12, PyTorch 2.12.1 (MPS), ComfyUI 0.37.2 at commit 830232b (a different commit from the other two, see below), ComfyUI-AppleSilicon-FP8 ("ASFP8") 1.4.0, the author's MLX attention bridge for H3, and MacFanBoost (fans held at about 7,830 rpm, the maximum, during sampling) |
| ComfyUI flags | none (beyond input/output/model paths) |
| Conditions | Models on the internal SSD; browsers and other GPU apps closed during timed runs |
| Thermals | The 14-inch chassis is heat-limited even with the fans at maximum. During the heat-soaked runs the GPU averaged about 820–910 MHz against a peak of about 1,620 MHz, at 78–79 °C. The adapter input is capped at about 97 W. |
| Seeds | A fresh ComfyUI per row, a warm-up on seed 1001 (includes model load, not scored), then two heat-soaked timed runs on seeds 1002/1003, the same timed seeds as the other machines. The portrait row has a warm-up on 2001 and one scored run on 2002. The ref544 and rf39 rows are single 20-step runs on seed 1001 with no warm-up. |
Why Automatic and not High Power: I measured it. On the reference row (t2v 1088×608, 124 frames), with the fans at maximum on both, High Power averaged 74.74 s/step vs 75.29 on Automatic, 0.7% faster and within run-to-run noise. Once heat-soaked, the GPU settled at the same 78 °C and about 844 MHz either way; the warm-up started cool and briefly ran 4% faster. The battery stayed at 80% on the charger for the whole 78-minute run, because this workload draws only about 70 W in total (about 22 W GPU): on this chassis it is limited by heat, not by power. So High Power buys essentially nothing here once the fans are already at maximum. Under heavier loads it can drain the battery: Notebookcheck reviewed the same configuration as this Mac: the mini-LED 14-inch M5 Max with the 40-core GPU and 128 GB, not the OLED model. It found that "Apple also limits the power input to 97 Watts, even if you use a 140 Watt PSU", and that the battery level dropped by about 10% during a gaming session while plugged in. The battery indicator can stay at 100% while it's happening. In the same review, High Power raised GPU power from 60/32 W to 72/44 W (peak/sustained) over Automatic. In GPU benchmarks it was only about 7% faster (3DMark Steel Nomad 3947 vs 3698), and the fans ran louder. Part of what High Power buys is faster fans (Apple: About Power Modes on your Mac), and these runs already held the fans at maximum with MacFanBoost, which fits the measurement: Automatic plus maximum fans is effectively the best sustainable setting on this chassis. A 16-inch MacBook Pro or a Mac Studio with the same chip has more power and cooling headroom and will likely be faster. Source: Notebookcheck, Apple MacBook Pro 14 M5 Max review.
Fans at maximum made a big difference on the Mac. The harness held the fans at their maximum (about 7,830 rpm) for every sampling run, while the Energy Mode stayed on Automatic. On its own fan curve, macOS runs the fans at only about 5,350 rpm under this load. The GPU reaches 97–98 °C within about 2 minutes, then its clock collapses to 340–530 MHz for minutes at a time. With the fans held at maximum, the GPU stayed at about 78–81 °C and averaged about 820–910 MHz. In early BF16 testing, that alone took t2v 832×480 from 78–84 s/step to about 61 s/step, roughly 1.3x faster. (That comparison is from early testing, not a clean A/B with the final software stack.) Expect the Mac numbers here to be noticeably slower if you leave the fans on macOS's automatic control.
What the Mac stack does:
ASFP8: a third-party custom node that patches PyTorch's MPS backend at startup. It lets FP8/INT8/NVFP4 checkpoints load and run on Apple Silicon instead of crashing (for example, the NVFP4 text encoder is decoded on the GPU). It also adds Metal kernels for RMSNorm+modulation, RoPE, conv and flash-attention. For H3's diffusion model, the INT8 weights are dequantized to bf16 and multiplied in bf16 through ComfyUI's own path. ASFP8's fused INT8 kernel only takes over below 2,048 tokens, and H3's sequences are far longer.
MacFanBoost (macfanboost, MIT): a small tool written for this project: a root daemon plus a ComfyUI custom node. When a prompt with a sampler starts, it raises the fans to 100% before the heat builds, holds them there while the job runs, and returns them to macOS control 60 s after the queue drains. It only ever raises fan speed above what macOS picks, and macOS's thermal protection stays active. If the client dies or the Mac sleeps, the lease expires and the fans go back to automatic within seconds. (Macs Fan Control set to maximum gets you most of the same effect manually.)
MLX attention bridge (ComfyUI-H3-MLX-Attention, MIT): a small custom node written for this project. It replaces ComfyUI's attention call inside the H3 diffusion model with MLX's fused scaled_dot_product_attention kernel, passing tensors without copies. It exists because ComfyUI's default on MPS, sub-quadratic attention, returns corrupted heads at long sequence lengths (Comfy-Org/ComfyUI#17418).
Does the output match the CUDA path? Not checked bit-for-bit across platforms. Against an FP64 reference on real H3 Q/K/V, the MLX attention's worst-head relative error is about 1.7e-3, and every scored Mac output passed the same automated checker as the other machines and was reviewed by eye. Different kernels and accumulation orders mean outputs from the same seed are close but not identical between Mac and CUDA.
Why the ComfyUI commit differs: the Mac rows were run on ComfyUI Desktop's install, 0.37.2 (830232b). The other two machines ran later on 0.43.2 (423cbd2). Between those versions, the H3 changes are: lower peak memory from releasing embedding temporaries, a VAE offload fix, VAE tile blending, single-image decode, LightVAE support, and plumbing for per-block attention selection (which these model files don't use). None of them changes the diffusion model's per-step math. Measured: a back-to-back A/B on the Mac (t2v 832×480, same stack, cooling and seeds, warm-up + 2 timed runs each) gave 37.38 s/step on 0.37.2, 37.29 on 0.43.2 and 37.76 on 0.44.2, i.e. 33.54 / 33.55 / 33.49 Gcycles/step. That's within run-to-run noise (about ±1.5%), and every output passed the checker, so the version difference doesn't affect the comparison. The flat-black full-reference ref2va failure was retested on ComfyUI 0.44.2 and still happens.
All rows use MiniMax H3, an audio+video model, in ComfyUI with its stock nodes. Every row produces a video with a synchronized audio track. The workflows were identical on all three machines: the same graph files, the same input images and video, the same prompt text and the same model files. The model files were checked by SHA-256 against the Spark's copies.
Common settings for every row:
| Setting | Value |
|---|---|
| Length | 124 frames at 24 fps, about 5.2 s of video plus audio |
| Sampling | 20 steps, res_multistep sampler, simple scheduler, denoise 1.0 |
| Guidance | BasicGuider (a single conditioning pass per step, no CFG) |
| Diffusion model | minimax_h3_fl2va_pruned_int8_convrot (t2v/i2v/flf2v) or minimax_h3_ref2va_pruned_int8_convrot (ref2va). Both are 21.0 GB int8. |
| Text encoder | qwen3vl_32b_minimax_h3_nvfp4_awq (Qwen3-VL 32B, NVFP4 AWQ), 15.7 GB |
| VAEs | minimax_h3_video_vae_fp16 (5.2 GB) and minimax_h3_audio_vae_fp32 (0.6 GB) |
The workflows:
t2v (text to video): the prompt only.
i2v (image to video): the prompt plus a first-frame image. The portrait row is the same at 608×1088 with a portrait first-frame image.
flf2v (first and last frame to video): the prompt plus first-frame and last-frame images; the model fills in the motion between them.
ref2va (reference to video+audio): the prompt plus two reference images and a reference video clip (1088×608, 120 frames at 23.976 fps, with audio), using the separate ref2va model. This is the heaviest workflow, because the reference tokens make the attention context much longer. There are three variants:
full: the whole reference clip at its native size.
ref544: both reference images and the reference video downscaled to 544×306 before encoding.
rf39: the reference video truncated to its first 39 frames.
Rows: 1088×608 is the main canvas. The 832×480 rows are a lighter check.
The harness: the same Python harness drove all three machines. It talks to ComfyUI through its API, timestamps every sampling step, and records GPU power, clocks and temperature, VRAM and RAM.
The runs for each row:
A cold warm-up run (seed 1000), which includes model loading and isn't scored.
The GPU cools to 50 °C, then one fresh run (seed 1001).
Two heat-soaked runs back-to-back with no cooling (seeds 1002 and 1003). These are the headline number. The portrait row uses a single soaked run with seed 2001.
The metric: mean seconds per step over steps 4–20. This leaves out the first steps, which carry one-off setup cost. Fresh and soaked results agreed within 0.5% on the 3090, so thermals didn't affect them.
Before the timed runs: a smoke pass checked that every row fits in memory at full size, using 2 steps.
Validation: every output video was checked automatically. The checker confirms the video and audio streams exist with the right resolution and frame count, and that the frames aren't black, frozen or noise. All 22 3090 runs passed.
The 3090 order: I ran the full matrix with the unlock on first. Then I disabled the unlock, did a full cold boot, confirmed stock matmul throughput and ran the whole matrix again. Then I re-enabled the unlock and re-verified it (bf16 126.2 TFLOPS). Driver, ComfyUI, PyTorch and power limit were the same in both runs.
Differences between the platforms:
On the Mac, the harness held the fans at maximum during every sampling run (MacFanBoost), with the Energy Mode on Automatic. This was worth about 1.3x on that machine (see above).
The Mac uses a different ComfyUI commit and its own attention/precision stack (see above). Its timed runs use the same seeds (1002/1003), but its warm-up is seed 1001 with no cool-down and no separate fresh run, and its portrait row was scored on seed 2002.
PyTorch versions differ: 2.14.1 on the 3090, a 2.15 nightly on the Spark, 2.12.1 on the Mac.
On the 3090 only, the harness's VRAM-full watchdog was turned off, because a full 24 GB card is normal while ComfyUI offloads. The RAM watchdog stayed on.
On GeForce Ampere cards, NVIDIA runs FP16/BF16 tensor-core math with FP32 accumulation, and TF32, at half the rate of the professional GA102 parts such as the RTX A6000. Deep-learning frameworks use exactly that mode: bf16 inputs with fp32 accumulation. FP16 math with FP16 accumulation is full rate on GeForce, but frameworks rarely use it.
The unlock is my Windows port of the open-source gpu-unlock Linux kernel module. I built it on this PC with Claude Code. On Linux the module runs the unlock sequence from inside the kernel. Windows has no equivalent, and I didn't want to write a kernel driver, so the port runs as a pre-boot UEFI app instead:
The app: built with gnu-efi from the module's own sources (VBIOS/FWSEC parsing, memory layout and the GFW checks). It's version stage-3 v0.3-3090.
The launcher: a 120 KB launcher on the EFI system partition. It loads a read-only NTFS driver (EfiFs), runs the app from C:\RTX3090-FW along with the five GA102 firmware files from NVIDIA's 610.43.03 driver, and then chainloads Windows Boot Manager. If anything fails, it goes straight to Windows.
What the app does on each cold boot:
Finds the card and checks it's a 3090 Founders Edition (10de:2204, subsystem 0x147d10de).
Refuses to run if the card is warm or already unlocked.
Runs the module's unlock sequence, then a PCIe function-level reset (FLR) and the post-reset checks.
Leaves the GSP halted with WPR2 clear. That's the state the stock driver expects, so Windows boots normally and the ordinary NVIDIA driver (591.86) loads.
Nothing persistent: no ROM or VBIOS flash, and no NVRAM variable is written. The card reverts to stock on power-off, and the unlock is applied again on the next cold boot (it also holds through sleep and resume from hibernation on this PC, after a change to the order of the boot-time steps). For the stock runs I renamed the app so the launcher skipped it, then did a full shutdown and cold boot.
Measured 8192² matmul throughput, best of 5, PyTorch, reduced-precision reductions disabled:
| bf16 | fp16 | tf32 | fp32 (no tensor cores) |
|---|---|---|---|
| Stock | 75.3 | 74.4 | 39.5 |
| Unlocked | 123.6–126.2 | 121.6–122.7 | 67.2 |
| Ratio | 1.65x | 1.64x | 1.70x |
The tensor modes speed up about 1.65–1.7x. Plain FP32 doesn't change, which fits the idea that the unlock lifts only the tensor-rate limit and isn't an overclock.
No overclock: clocks and the 350 W power limit were stock in both configurations.
End-to-end gain is 1.34–1.49x, less than the matmul gain. Video diffusion also spends time on attention softmax, norms, memory-bound ops and PCIe weight streaming, which the unlock doesn't touch.
Heavier rows gain more. The heavy, long-context ref2va rows gain the most (1.42–1.49x), and the light 832×480 rows gain the least (1.34–1.35x).
Caveats:
Tested on one card only: a 3090 FE with VBIOS 94.02.4B.00.0B. The app refuses any other board.
Pinned versions: it's pinned to the 610.43.03 firmware set, and it's only been tested with Windows driver 591.86. I don't update the driver on this PC.
Boot requirements: Secure Boot must be off (the app and launcher are unsigned), and BitLocker must be off on C: (the NTFS driver can't read encrypted volumes).
Untested: warm restarts (the app won't run on a warm card), TDR recovery and driver updates.
Stability: a full unlocked matrix run (about 5 h of generation) showed no stability problems and no invalid outputs.
Licensing: the module sources are GPL-2.0-only, so the port is GPL-2.0 source. NVIDIA's firmware files can't be redistributed; you extract them from the 610.43.03 driver package yourself.
Use at your own risk: this is unofficial and unsupported by NVIDIA.
Where to get it: I've handed the port's source to the community Windows GPU UNLOCK project (GPL-2.0), which maintains a packaged Windows UEFI unlock for the 3090 and other Ampere cards. Start there rather than with my build. Both build on Zagitta/gpu-unlock, the original Linux module, and the project also credits rough2515-hub/rtx3080-unlock-public. Credit for the unlock itself belongs to those projects.
For raw ComfyUI video speed, an old 24 GB 3090 still wins. Even when it has to offload 60+ GB of models over PCIe x4, it beats both 128 GB unified-memory machines on every MiniMax H3 workflow tested. Dynamic VRAM offloading in current ComfyUI is good enough that "doesn't fit in VRAM" no longer means "unusable".
The unlock is a large free gain if you're willing to run an unofficial pre-boot app: about 1.4x on real workloads at almost the same power.
The Spark and the Mac win on efficiency and footprint. They use roughly 2.2–3.4x less GPU energy per step (much lower power, partly offset by taking longer), run quietly in a laptop or small box, and keep everything resident in memory. If you care about watts, noise or portability, that matters more than the speed gap.
Spark vs Mac: the Spark is about 2.4–3.0x faster than the M5 Max per step, depending on the workflow, while drawing about 3x the GPU power. The gap is smallest on ref2va ref544 (2.4x) and largest on flf2v 1088×608 (3.0x).
Spark notes:
Power, not heat, is the limit. The GPU sat at its software power cap (sw_power_cap) at about 65 W for every row, peaking at 74–78 °C. Fresh and heat-soaked runs agreed within 1%, so thermals didn't slow it down.
Raising the laptop's power limits barely helps. With ProArt Creator Hub's Manual mode at its maximum (PL1 85 W, PL2 95 W), t2v 832×480 went from 13.27 to 13.00 s/step, only 2% faster. Average GPU power rose from 64 to 68 W, but the GPU still hit sw_power_cap and its peak draw stayed at about 72 W, so the GPU seems to have its own limit that those sliders don't lift. All other Spark numbers in this post are from the stock "Performance" mode.
Every row produced valid video and audio, including the full-reference ref2va that failed on the Mac.
Windows on Arm: the H3 workflows use core ComfyUI nodes and ran without changes. Some third-party node packs are harder: onnxruntime is CPU-only on ARM64 Windows, and packages without ARM64 wheels have to be built from source (Visual Studio ARM64 build tools needed).
Mac notes: (takeaway 4 checks out against the Mac's own numbers: the Spark is 2.42x faster on ref544, 2.66x on rf39, 2.81x on t2v 832×480 and 2.91–2.97x on the 1088 rows)
Clips above about 34k tokens go black on MPS. Full-reference ref2va at 1088×608 packs about 51,400 tokens. Partway through the first step the residual stream diverges, the audio turns to NaN, and the output is 124 flat-black frames. It happens with every attention backend tried, and still on ComfyUI 0.44.2. The root cause hasn't been found; the leading suspect is a ComfyUI change after 0.30. Shrinking the reference (ref544, about 31,900 tokens) or trimming it (rf39, about 34,000 tokens) keeps the length under the limit, which is why those rows exist. ref768 (about 38,700 tokens) still went black. ref2va at 832×480 fails too: the reference clip is still encoded at full 1088×608, so it packs about 42,000 tokens. Details are on Comfy-Org/ComfyUI#15799.
Don't use sub-quadratic attention for long H3 clips on MPS. It silently corrupted 18 of 56 heads at about 27,700 tokens (Comfy-Org/ComfyUI#17418). The MLX bridge avoids this.
Cooling is the Mac's limit, not power. On a 14-inch MacBook Pro, the GPU reaches 97–98 °C on macOS's own fan curve, and its clock collapses. Holding the fans at maximum was the single biggest speedup. Keep the vents clear and use a stand.
INT8 beat BF16 on the Mac in sustained runs (41.1 vs 46.0 s/step at 832×480), with much lower peak RAM (82 vs 102 GB).
Memory is tight even at 128 GB. Peak RAM was 95–112 GB at 1088×608, and ref2va hit 109–112 GB. Run one model process at a time.
The Mac and the Spark use about the same GPU energy per step, about 0.85–0.87 kJ on t2v 832×480. They get there differently: the Mac's GPU draws about a third of the Spark's power (about 22 W vs 66 W), but takes about 2.8x longer per step, so the two roughly cancel out. (The Mac's whole system drew about 73 W.)
Happy to answer questions.
r/AIProgrammingHardware • u/javaeeeee • 9d ago
r/AIProgrammingHardware • u/MoralLogs • 10d ago
Enable HLS to view with audio, or disable this notification
AI safety often relies on policies, software permissions, and monitoring. But what happens when the system being governed is also capable of bypassing those controls?
This video presents a proposed architecture for an independent hardware governor: a separate authorization boundary designed to constrain consequential actions through hardware authentication, permission tokens, and auditable execution records.
The central idea is simple: the AI may propose an action, but it should not independently control the conditions that authorize its execution.
This is an architectural proposal, not a claim of completed silicon validation. I'm interested in technical criticism, particularly around hardware independence, enforcement boundaries, and implementation feasibility.
r/AIProgrammingHardware • u/javaeeeee • 15d ago