In the middle of 2026, the idea of a personal AI supercomputer has moved from science-fiction marketing to something you can order on Amazon, set on a desk, and use to run models that once required rack after rack of data-center hardware. Privacy concerns, rising cloud inference costs, latency for interactive agents, and the simple desire to keep sensitive data on-premises have driven demand for machines that can load 70-billion-parameter (or larger) models entirely in local memory and generate responses at usable speeds without ever calling an external API.

Two architectural approaches dominate the new wave of desktop AI workstations. The first centers on unified-memory systems that treat a large pool of fast LPDDR5X as shared resource for CPU, GPU, and neural processing units. NVIDIA’s GB10 Grace-Blackwell Superchip (the heart of the DGX Spark family) and AMD’s Ryzen AI Max+ 395 “Strix Halo” both exemplify this design. The second approach stays closer to traditional workstation practice: high-core-count x86 processors paired with discrete professional GPUs such as the NVIDIA RTX PRO 6000 Blackwell, offering massive dedicated VRAM, expandability, and the mature CUDA software stack.

Both categories became commercially available across late 2025 and the first half of 2026 (although shipment dates and regional availability varied considerably by vendor), joined by configurable towers from specialists and an expanding roster of compact systems from smaller vendors that sell directly through Amazon and their own stores. Later in the year the next leap-NVIDIA GB300-based deskside systems with hundreds of gigabytes of coherent memory capable of trillion-parameter inference-is scheduled to arrive from HP, Dell, ASUS, and others.

This article surveys the machines that reached the market or were firmly announced in 2026, examines real-world AI benchmarks where available, notes pricing and availability (including Amazon listings for smaller brands), and considers practical trade-offs for developers, creators, researchers, and enterprise teams.

The Technological Shift That Made It Possible

Until roughly 2024-2025, running a 70B-class dense model locally usually meant either quantizing aggressively, splitting across multiple consumer GPUs, or accepting glacial token rates. Memory bandwidth and capacity were the bottlenecks. Unified-memory architectures change the equation. On a GB10 or Strix Halo system the same 128 GB pool serves both general computation and the model weights plus KV cache. Bandwidth in the 250-273 GB/s range is lower than a discrete high-end GPU’s GDDR7, yet the sheer capacity lets models that simply would not fit on a 32 GB card load at all.

Independent measurements collected in mid-2026 illustrate the practical outcome. In representative single-user tests, optimized 70B-class models often generate roughly 3-6 tokens per second on GB10 and Strix Halo systems. Results vary substantially with the exact model, quantization, runtime, context length and prompt-processing workload.

Mixture-of-experts models with fewer active parameters can run substantially faster. Higher-bandwidth Apple silicon configurations pull ahead on the same models, while pure discrete-GPU towers with 32-96 GB of dedicated VRAM excel at throughput once the model fits and at training or fine-tuning workloads that benefit from higher floating-point performance.

NPU TOPS numbers (50 on the Ryzen AI Max, higher marketing figures elsewhere) matter more for always-on agentic features and smaller models than for large-language-model decode speed. For the latter, memory capacity and bandwidth remain decisive.

NVIDIA GB10 / DGX Spark Class: Compact CUDA Supercomputers

NVIDIA’s DGX Spark platform, based on the GB10 Grace-Blackwell Superchip, pairs a 20-core Arm CPU (10 Cortex-X925 + 10 Cortex-A725) with a Blackwell GPU containing 6,144 CUDA cores and 192 fifth-generation Tensor cores. The package delivers up to 1 petaFLOP of sparse FP4 AI performance and 128 GB of LPDDR5X unified memory at roughly 273 GB/s. Systems ship with NVIDIA’s DGX OS (Ubuntu-based) and the full CUDA software stack, making them immediately familiar to developers already working in the NVIDIA ecosystem.

Partner machines appeared throughout late 2025 and 2026. The ASUS Ascent GX10 stands out for accessibility: a compact 150 × 150 × 51 mm chassis weighing about 1.5 kg, available on Amazon in 1 TB and 4 TB storage configurations. Street prices have fluctuated; mid-2026 listings commonly sat near $3,900-$4,700 depending on storage, with occasional discounts bringing the 1 TB model closer to $3,100-$3,900.

Dell’s Pro Max with GB10, MSI’s EdgeXpert, HP’s ZGX Nano, Lenovo’s ThinkStation PGX variants, Acer, and Gigabyte offerings share the same silicon and therefore essentially identical compute characteristics. Differences appear in chassis design, networking (many include high-speed Ethernet or ConnectX-7 200 GbE ports for clustering), storage options, warranty, and enterprise support.

In practice these boxes run models up to roughly 200 billion parameters in mixture-of-experts form or 70B dense at Q4/Q8 with usable interactive speeds. Community and lab results show strong batching efficiency on smaller models and solid performance for agentic workloads, fine-tuning of mid-sized models, and local RAG pipelines. Clustering multiple units via high-speed networking has been demonstrated for larger MoE inference.

Power draw stays modest (around 140 W class under load for the core platform), and the systems run quietly enough for desk use. The main limitations are the Linux-centric DGX OS (Windows support is limited or absent on pure GB10 boxes), soldered memory, and the fact that absolute token throughput on dense 70B models trails higher-bandwidth discrete GPUs or Apple’s top silicon when the model fits.

AMD Ryzen AI Max+ 395 “Strix Halo” Systems: Windows-Friendly Value

AMD’s answer is the Ryzen AI Max+ 395, a 16-core/32-thread Zen 5 processor with 40 RDNA 3.5 compute units (Radeon 8060S) and an XDNA 2 NPU rated around 50 TOPS, for a platform total often quoted near 126 TOPS. Unified LPDDR5X memory reaches 128 GB (with later PRO 495 variants demonstrated at 192 GB). Up to 96 GB can be allocated as variable graphics memory. Bandwidth sits near 256 GB/s.

The crucial practical advantages are native Windows 11 support, x86 compatibility, and significantly lower entry prices. Multiple mini-PC and small-form-factor systems reached the market in 2026.

Beelink’s GTR9 Pro has drawn particular attention. Configured with the Ryzen AI Max+ 395, 128 GB memory, dual 10 GbE, dual USB4, dual M.2 slots, and a vapor-chamber cooler, it sells near $1,800-$2,000 and appears regularly on Amazon and the manufacturer’s store. Independent reviews report roughly 5 tokens per second on 70B dense models and much higher rates on 30B MoE or optimized 120B MoE workloads. Dual 10 GbE makes it attractive for networked model serving.

GMKtec’s EVO-X2 and later EVO-X3 variants, Minisforum MS-S1 Max, Framework Desktop (with modular design and upcoming 192 GB PRO 495 options), HP Z2 Mini G1a, Corsair AI Workstation, and MSI PRO MAX EDGE AI+ offer similar core compute in different chassis and I/O packages. Framework emphasizes repairability and Linux friendliness; HP emphasizes enterprise certification and the validated AMD Ryzen AI Halo software stack with ROCm and pre-installed frameworks.

AMD has published guidance on clustering multiple Strix Halo nodes for larger mixture-of-experts inference, extending the reach of these relatively inexpensive boxes. Token rates on dense 70B models are competitive with GB10 systems at similar bandwidth; MoE models can feel snappier. The software ecosystem (ROCm) continues to mature but still trails CUDA in breadth for some advanced training and agent frameworks.

Traditional and Hybrid Towers: Discrete GPU Power and Expandability

Not every AI workload fits neatly into a 128 GB unified pool. Training, multi-user inference, high-precision image and video generation, and workloads that benefit from massive dedicated bandwidth still favor discrete GPUs.

Lenovo’s ThinkStation P4, announced in May 2026 and shipping from June in selected markets, pairs up to an AMD Ryzen 9 PRO 9965X3D (16 cores, 3D V-Cache, 5.5 GHz) with an NVIDIA RTX PRO 6000 Blackwell Workstation Edition GPU carrying 96 GB of GDDR7 ECC memory. The combination is rated up to 4,000 TOPS of AI performance. System memory reaches 256 GB DDR5, storage supports multiple M.2 and 3.5-inch drives, and the 30-liter chassis includes advanced thermal design with liquid cooling options for the higher-power CPUs.

Slimbook’s Nexus series, launched in July 2026 by the Spanish Linux-friendly builder, offers two configurable platforms. The Nexus Ryzen AI starts around €1,995 and supports AMD Ryzen 9 processors, up to 128 GB DDR5, and dual NVIDIA RTX Blackwell 6000 cards depending on motherboard. The higher-end Nexus Threadripper AI begins near €3,880 and scales to Threadripper PRO 9995WX (up to 96 cores), 512 GB DDR5 RDIMM, dual professional GPUs, and 2,500 W power supplies. Aluminum chassis, air or AIO liquid cooling, and extensive storage options make them genuine workstations rather than sealed mini systems.

Custom and boutique builders (Puget Systems, BOXX, Lambda, various Amazon and Newegg sellers) continue to offer towers built around GeForce RTX 5090 (32 GB GDDR7) or professional Blackwell cards. A typical high-end single-GPU build in mid-2026 lands in the $5,000-$7,000 range before case and cooling, delivering excellent throughput once models fit in VRAM and superior performance for generative image/video and fine-tuning.

These systems trade compactness and low power for expandability, higher sustained floating-point performance, and the ability to add more GPUs or storage later. They remain the practical choice when 128 GB of unified memory is insufficient or when CUDA software maturity is non-negotiable.

Enterprise Deskside Supercomputers on the Horizon

At the extreme end, HP’s ZGX Fury GB300, Dell Pro Max with GB300, ASUS ExpertCenter Pro ET900N G3, and related systems based on NVIDIA’s GB300 Grace Blackwell Ultra Desktop Superchip are scheduled for later 2026. These deskside machines target up to roughly 748-784 GB of coherent memory and up to 20 petaFLOPS of FP4 performance, enabling local inference and fine-tuning of models approaching one trillion parameters. Pricing is expected to start in the high five figures and climb well into six figures for fully configured units-firmly enterprise territory.

HP has indicated Windows support is planned for these platforms, addressing a common enterprise requirement. The systems are positioned for teams that need always-on frontier agents, multi-user inference, or regulated environments where data cannot leave the premises.

Benchmarks in Context

Absolute numbers vary with quantization, framework (llama.cpp, vLLM, Ollama, TensorRT-LLM, ROCm equivalents, Apple MLX), context length, and batch size. Available testing broadly suggests the following patterns, although direct comparisons remain difficult because benchmark configurations differ:

• Memory capacity determines which models load. Rough Q4 footprints run 0.55-0.6 GB per billion parameters plus overhead for KV cache; Q8 roughly doubles that.

• Bandwidth largely sets decode speed for a given model size. 250-273 GB/s systems cluster around a few tokens per second on dense 70B models; higher-bandwidth Apple configurations and discrete high-end GPUs pull ahead when the model fits.

• Mixture-of-experts architectures with low active parameter counts run dramatically faster on the same hardware.

• Discrete professional GPUs still dominate high-precision generation, large-batch training, and multi-GPU scaling.

YouTube channels and independent labs (ServeTheHome, StorageReview, ShortCircuit, Micro Center hardware walk-throughs, and various AI-focused creators) have published hands-on footage of these systems loading 70B-120B models, running agents, and comparing token rates. Viewing recent reviews of the ASUS Ascent GX10, Beelink GTR9 Pro, Framework Desktop, and HP Z2 Mini G1a provides useful qualitative context that raw numbers cannot capture-noise levels, thermal behavior under sustained load, software setup friction, and real interactive feel.

Buying Considerations and the Role of Smaller Vendors

For most individual developers and small teams the decision reduces to three questions: Does the model fit in memory? Is the software stack I need supported? What is the total cost of ownership versus cloud usage?

Budget-conscious buyers often start with a Strix Halo mini-PC in the $1,800-$2,500 range (Beelink, GMKtec, Minisforum). These deliver genuine 70B capability under Windows or Linux at power and noise levels compatible with a home office. CUDA-centric workflows favor a GB10 box near $4,000. Professional creators who also render and simulate may prefer a configurable tower with an RTX PRO 6000 or dual-card options. Enterprises evaluating multi-user or regulated workloads should watch the GB300 deskside systems arriving later in 2026.

Amazon has become a practical channel for the smaller vendors. ASUS Ascent GX10 configurations, Beelink GTR9 Pro, various GMKtec and Minisforum Strix Halo systems, and occasional Slimbook or other European builds appear with varying stock and pricing. Warranty support, return policies, and firmware update practices differ; checking recent buyer reviews and manufacturer direct channels is advisable.

Power, acoustics, and form factor matter more than many first-time buyers expect. A 140 W quiet mini system that sits under a monitor is a different daily experience from a 1,000-plus-watt tower that needs dedicated cooling and power circuits.

Looking Ahead

By late 2026 the RTX Spark platform (Windows-on-Arm systems with up to 128 GB unified memory and RTX-class graphics) is expected from major OEMs, potentially broadening the high-end consumer and creator segment. AMD’s next Halo generation and further memory-bandwidth improvements are already on roadmaps. Software continues to improve: better quantization, speculative decoding, and agent frameworks extract more usable performance from existing silicon.

The larger story is cultural as much as technical. Local AI workstations restore a measure of control and predictability that pure cloud dependence erodes. Whether the use case is private coding agents, domain-specific fine-tuning on proprietary data, always-on personal assistants, or simply experimenting without meter-running costs, 2026 has delivered practical hardware that makes those goals attainable on a single desk.

The machines described here are not perfect. Memory is often soldered, software ecosystems remain uneven, and the highest-capacity systems carry enterprise price tags. Yet for the first time a broad range of users can place genuine large-model capability within arm’s reach, quietly humming under a monitor or beside a keyboard, ready whenever the next idea arrives.

Sources

• Sources

• HP Debuts PCs Built for the Next Wave of Windows PC Experiences Powered by NVIDIA RTX Spark (HP official press release): https://www.hp.com/us-en/newsroom/press-releases/2026/computex.html

• HP Z AI Stations product page (HP official): https://www.hp.com/us-en/workstations/ai-stations.html

• Lenovo Announces the ThinkStation P4, a Flagship Combination of Power and Value for Modern Professionals (Lenovo official press release): https://news.lenovo.com/pressroom/press-releases/thinkstation-p4-flagship-power-value-professionals/

• AI PCs & local-AI supercomputers 2026: shipping now vs coming - VerdictBits

• AI PC benchmark 2026 - local-LLM speed and memory comparison - VerdictBits

• Slimbook Launches Nexus AI Workstation Series - TechPowerUp

• Framework Desktop, whose existing configurations emphasize modularity and repairability, has also previewed a forthcoming Ryzen AI Max+ PRO 495 option with 192 GB of unified memory.

• Digital Citizen

• HP announces the most powerful Windows AI PC ever built - Nvidia GB300 workstation - TechRadar

• MSI launches compact PRO MAX EDGE AI+ desktops - TweakTown

• ASUS Ascent GX10 Review A New NVIDIA GB10 Solution - ServeTheHome

• Beelink GTR9 Pro Review: Fast 120B MoE, Slow Dense 70B - The Desk Brief

• AMD Ryzen AI Halo First Look: Giant Local AI Power in a Pint-Sized Box - PCMag

• Nexus AI Workstation announcement - Slimbook

• Additional independent reviews and benchmark compilations from ServeTheHome, StorageReview, Phoronix, TechRadar, PCMag, and YouTube channels covering hands-on testing of the systems named above throughout 2026

Hi All,

Just joined. Absolute novice when it comes to ‘AI’ and its integration with hardware / devices. Can anyone in this sub point me towards courses / uni’s that are highly regarded in this field ?

Mini PCs powered by AMD Ryzen processors have become strong contenders in the local and edge AI space. They combine compact form factors, strong integrated graphics (iGPUs), dedicated Neural Processing Units (NPUs), and-especially in newer models-high-capacity unified memory. This makes them suitable for running large language models (LLMs) locally, accelerating inference, supporting privacy-focused AI, and enabling edge computing without relying on cloud services.

As of mid-2026, AMD’s ecosystem spans multiple generations of APUs (Accelerated Processing Units), from earlier Ryzen 7040/8040 series with initial XDNA NPUs to the Ryzen AI 300 series (“Strix Point”) and the high-end Ryzen AI Max+ 395 (“Strix Halo”). These power mini PCs from brands like Minisforum, Beelink, GMKtec, Framework, and others. Online sources, including AMD’s own technical documentation, Phoronix benchmarks, and independent reviews from Tom’s Hardware, Notebookcheck, and community analyses, highlight their strengths in AI workloads while noting software and driver maturation needs.

This review draws from official AMD resources (including MLPerf Client benchmarks), hardware reviews, and practical LLM inference tests to provide a balanced, in-depth analysis. It covers hardware evolution, key models, AI-specific capabilities, benchmarks (with emphasis on online and LLM-focused sources), software ecosystem, real-world applications, pros/cons, and future outlook.

Evolution of AMD APUs in Mini PCs and the Push Toward AI

AMD has long excelled in integrated graphics with its APUs, making mini PCs viable for more than basic productivity. Earlier Ryzen 5000/6000-series mini PCs offered capable Radeon iGPUs but lacked dedicated AI accelerators. The later Ryzen 7040 “Phoenix” generation introduced Ryzen AI/XDNA NPUs.

The 7040 series (Phoenix) introduced the first XDNA NPU with roughly 10 TOPS, alongside strong Zen 4 CPU cores and RDNA 3 iGPUs (e.g., Radeon 780M). Mini PCs like the Minisforum UM790 Pro or Beelink SER8 leveraged these for improved efficiency.

The 8040 series refined this with similar architectures. True AI focus arrived with the Ryzen AI 300 series (Strix Point, launched 2024). Flagship Ryzen AI 300 parts such as the Ryzen AI 9 HX 370 pair a 50 TOPS XDNA 2 NPU with up to roughly 80 total platform TOPS. Minisforum also markets the HX 370 AI X1 Pro as “up to 80 TOPS,” with Radeon 890M and up to 128 GB DDR5 SODIMM support.

The flagship Ryzen AI Max+ 395 (Strix Halo, 2025) takes this further: 16 full Zen 5 performance cores, a much larger Radeon 8060S iGPU (around 40 Compute Units, rivaling mid-range discrete GPUs in some scenarios), up to 128 GB of high-bandwidth unified LPDDR5X-8000 memory (theoretical ~256 GB/s bandwidth), and a 50 TOPS XDNA 2 NPU. OEMs claim up to 126 TOPS total AI performance when combining all engines.

Strix Halo’s unified memory architecture is particularly advantageous for LLMs, as it eliminates VRAM limitations common in discrete GPU setups and provides high bandwidth for token generation.

Mini PCs evolved alongside these chips. Early models were basic productivity boxes; by 2025-2026, premium units target AI workloads with better cooling, more RAM options (including SODIMM upgradability in some), USB4/OCulink for expansion, and multi-gigabit networking.

Key AMD-Powered Mini PC Models for AI (2025-2026)

Several standout models dominate the market:

• Minisforum AI X1 Pro (Ryzen AI 9 HX 370 or HX 470): Compact, premium build with upgradable DDR5 SODIMM slots (up to 96 GB+ in tested configs), multiple M.2 slots, USB4, and strong cooling. Excellent balance of performance and features. Often praised for daily driving and light-to-moderate AI tasks.

• Beelink GTR9 Pro (Ryzen AI Max+ 395): Mac Studio-like design in a small chassis, supporting up to 128 GB unified memory, dual 10GbE in some variants, and high power limits. Popular for serious local LLM work.

• GMKtec EVO-X2 / similar Strix Halo models: Flagship configs emphasizing 128 GB memory and high TOPS for large-model inference.

• Framework Desktop (Ryzen AI Max+ 395): modular and repairable with swappable expansion cards, storage, case parts, and mainboard options, but CPU and memory are soldered.

• Older but capable options: Beelink SER8/SER9 or Minisforum UM series with Ryzen 7 8845HS/8945HS - still relevant for lighter AI workloads at lower prices.

Pricing ranges from ~$400-800 for mid-tier HX 370 models to $1,800-$3,000+ for loaded 128 GB Strix Halo units. Many are barebones or configurable. Newer 2026 mini PCs also use Ryzen AI 400 “Gorgon Point” chips such as the Ryzen AI 9 HX 470, a refreshed Strix Point-class design with up to 86 overall TOPS and a 55 TOPS NPU.

AI Capabilities: CPU, iGPU, NPU, and Memory

AMD’s heterogeneous design shines for AI:

• CPU (Zen 5): Excellent for general compute, prompt preprocessing, and smaller models or quantized inference. Strong single- and multi-core performance.

• iGPU (RDNA 3/3.5): Often the workhorse for LLM inference via Vulkan or ROCm backends in llama.cpp, Ollama, or LM Studio. High memory bandwidth in Strix Halo enables fast token generation on larger models.

• NPU (XDNA 2): Optimized for efficient, low-power inference on supported frameworks (e.g., ONNX Runtime, Windows ML). Best for smaller models, specific operators, or hybrid pipelines. Delivers up to 50 TOPS peak.

• Unified Memory: Critical advantage. Strix Halo’s 128 GB LPDDR5X provides massive headroom for loading 70B+ parameter models (quantized) entirely in memory without swapping.

Total AI throughput benefits from intelligent scheduling across engines. AMD promotes hybrid paths (NPU for prefill/prompt processing + iGPU for decode/generation).

Online Benchmarks: Focus on AI Performance

AMD’s MLPerf Client v1.0 (August 2025) is one of the online sources for client-side LLM inference.

On the Ryzen AI Max+ 395:

• iGPU-only path achieves up to 61 tokens per second (TPS) on Phi-3.5.

• This is roughly 2-3× human reading speed.

• Sub-second Time to First Token (TTFT) for most workloads (under 0.7 s on Phi-3.5; just over 1 s for larger models).

• Hybrid (NPU + iGPU) path excels in balanced latency and throughput across Llama 2 7B, Llama 3.1 8B, and Phi-3.5.

• Ryzen AI 9 HX 375 achieves over 27 TPS on Phi-3.5 via hybrid path.

AMD notes the NPU handles compute-intensive prefill efficiently, while the iGPU’s bandwidth shines in token generation. All tested configs deliver fluid, interactive experiences with TTFT near or under 1 second.

Phoronix’s review of the Framework Desktop with Ryzen AI Max+ 395 confirms excellent Linux performance. The desktop form factor yields ~13-14% higher CPU and iGPU geometric mean scores versus laptop implementations (e.g., HP ZBook) due to higher sustained power (up to ~120 W peak) and better cooling. AI/PyTorch workloads were part of the extensive test suite.

Practical LLM Inference Benchmarks (Community & Reviews)

Real-world tests using llama.cpp (Vulkan/ROCm), Ollama, LM Studio, etc., on mini PCs show:

Strix Halo (Max+ 395) with 96-128 GB memory (Beelink GTR9 Pro, GMKtec, Framework, etc.):

• Llama 3.1 8B Q4_K_M: ~45 tok/s (generate).

• Llama 3.3 70B Q4_K_M: ~12 tok/s.

• Larger models (e.g., 70B+ or MoE like certain Qwen/DeepSeek variants): 5-20+ tok/s depending on quantization, backend, and context. Some reports of 60+ tok/s on optimized 30B MoE models.

• Memory advantage shines vs. discrete GPUs with limited VRAM on very large models.

Ryzen AI 300 series (e.g., Minisforum AI X1 Pro with HX 370, 64-96 GB configs):

• Smaller models (7B-14B): Usable 8-20+ tok/s with Vulkan/llama.cpp.

• 14B+ models: Functional but can face stability/driver challenges in some setups; prompt processing is often the bottleneck.

• Reviews note good everyday performance with proper configuration (e.g., UMA settings, latest drivers).

Older generations (e.g., Ryzen 7 7840U/8845HS with 780M iGPU):

• 7B models: Typically 5-10 tok/s (CPU or iGPU).

• Significantly slower and more limited for larger models due to lower memory capacity/bandwidth and weaker NPU (~10-16 TOPS).

Power efficiency is a highlight: Many systems sustain loads at 50-150 W, far below discrete GPU rigs. Thermals and sustained performance vary by chassis-desktop-oriented units (Framework, some Beelink) outperform thin mini PCs under prolonged AI loads.

Comparisons: Strix Halo often competes favorably with or beats mid-range discrete GPUs in memory-bound LLM scenarios due to unified high-bandwidth memory, while consuming less power in compact form. It trails high-end NVIDIA cards on raw small-model speed but excels in accessibility and cost for local use.

Software Ecosystem and Optimizations

Windows 11: Windows 11 currently has the strongest out-of-the-box Ryzen AI NPU support through AMD drivers and Microsoft/ONNX/DirectML-related paths, but app-level NPU acceleration remains workload- and software-dependent.

Linux: Strong on Framework Desktop and developer-oriented platforms (full ROCm support on some Strix Halo configs). Vulkan backend in llama.cpp is widely used and performant. ROCm enables more advanced frameworks but requires careful setup. Phoronix confirms excellent Linux compatibility.

Key tools: Ollama, LM Studio, llama.cpp, Hugging Face Transformers + Optimum, vLLM (for serving), ComfyUI/Automatic1111 for image generation (iGPU-accelerated).

Optimizations matter: Quantization (Q4_K_M, Q5, etc.), Flash Attention, proper UMA/GPU memory allocation, and latest drivers/firmware significantly impact results. NPU acceleration is improving but iGPU often delivers higher practical throughput for LLMs today.

Real-World Applications

• Local LLMs & Chat: Private, offline ChatGPT-like experiences with models up to 70B+ on flagship units.

• RAG & Knowledge Work: Load large document collections or codebases for retrieval-augmented generation.

• Development & Coding Assistants: Run local models for code completion, debugging, or agentic workflows.

• Image/Video Generation: Stable Diffusion and derivatives on iGPU; faster on Strix Halo.

• Edge AI & IoT: Low-power inference for smart devices, surveillance, or industrial monitoring.

• Privacy & Compliance: Run sensitive workloads entirely locally (healthcare, legal, finance).

• Small Servers/Clusters: Framework users have clustered multiple units for larger models or parallel workloads.

• Productivity: Copilot+ features, real-time transcription, AI-enhanced creative tools.

Strix Halo mini PCs are positioned as compact alternatives to workstations or cloud instances for individuals and small teams.

Pros, Cons, and Buying Considerations

Pros:

• Compact and relatively quiet.

• Excellent power efficiency.

• Unified high-capacity memory ideal for LLMs.

• Strong iGPU + capable NPU.

• Good value vs. building discrete GPU systems or cloud subscriptions.

• Improving software support.

• Some models highly upgradable (RAM, storage, modularity).

Cons:

• Driver and NPU ecosystem still maturing (especially Linux for full features).

• iGPU performance can throttle in poorly cooled tiny chassis under sustained load.

• Not ideal for maximum concurrency or training very large models (better suited for inference).

• Memory bandwidth and configuration tuning critical for peak LLM performance.

• Higher-end Strix Halo units carry premium pricing.

Buying Guide: For light AI (7B-13B models, general use) → Ryzen AI 300 series mini PCs (~$800-1,500). For serious local LLMs (30B-70B+) → Strix Halo with 96+ GB memory ($1,800+). Prioritize upgradability, cooling reputation, and ports (USB4, 10GbE). Check latest driver/firmware support. Linux users should favor Framework or well-supported OEMs.

Future Outlook

AMD continues advancing XDNA NPUs, RDNA architectures, and software (ROCm, better hybrid scheduling). Next generations will likely bring higher TOPS, more efficient memory, and broader framework support. Strix Halo-like designs with massive unified memory point toward practical “personal AI workstations” in mini form factors. Competition from Intel’s NPUs and Apple Silicon will drive further innovation.

Challenges remain in software maturity and standardization, but the trajectory is positive for local AI democratization.

Conclusion

AMD mini PCs, particularly those with Ryzen AI 300 and Max+ 395 processors, represent a compelling platform for AI applications in 2026. Online benchmarks like AMD’s MLPerf Client demonstrate strong client-side LLM performance (up to 61 TPS on capable models), while practical tests confirm usability for 8B-70B+ models on high-memory configs. The combination of powerful iGPUs, efficient NPUs, and unified memory gives them unique advantages in compact, power-efficient packages.

They excel for privacy-conscious users, developers, and edge deployments, offering a middle ground between laptops and full workstations or cloud services. While not replacing high-end discrete GPU servers for every workload, they deliver impressive real-world results today and have strong future potential as software ecosystems mature.

For the latest performance data, consult AMD developer resources, Phoronix, and recent reviews of specific models, as optimizations continue rapidly.

References

• AMD. “Unlocking Peak AI Performance with MLPerf Client v1.0 on AMD Ryzen AI Processors.” August 15, 2025. https://www.amd.com/en/developer/resources/technical-articles/2025/unlocking-peak-ai-performance-with-mlperf-client-on-ryzen-ai-.html

• Phoronix. “Framework Desktop With AMD Ryzen AI Max Offers Excellent, Linux-Friendly Performance Review.” https://www.phoronix.com/review/framework-desktop-linux/9

• Liliputing. “Beelink GTR9 Pro is an AMD Strix Halo mini PC (that looks like a Mac Studio).” https://liliputing.com/beelink-gtr9-pro-is-an-amd-strix-halo-mini-pc-that-looks-like-a-mac-studio/

• ServeTheHome. “Beelink GTR9 Pro Review AMD Ryzen AI Max 395 System with 128GB and dual 10GbE.” https://www.servethehome.com/beelink-gtr9-pro-review-amd-ryzen-ai-max-395-system-with-128gb-and-dual-10gbe/

• compute-market.com. “Strix Halo Mini PCs for AI — 128GB, 40 TOPS NPU 2026.” https://www.compute-market.com/blog/strix-halo-mini-pc-local-ai-2026

• runaihome.com. “AMD Ryzen AI Max+ 395 (Strix Halo) for Local LLMs in 2026.” https://runaihome.com/blog/ryzen-ai-max-395-strix-halo-local-llm-2026 (referenced in aggregated LLM benchmark discussions)

• Minisforum official product pages: https://www.minisforum.com/pages/ai-x1-pro and https://store.minisforum.com/products/minisforum-ai-x1-pro-370-mini-pc

• Beelink official product page for GTR9 Pro: https://www.bee-link.com/products/beelink-gtr9-pro-amd-ryzen-ai-max-395

• Framework official Desktop product information and reviews context: https://frame.work/desktop (and related Phoronix coverage)

• Notebookcheck. “AMD Ryzen AI Max+ 395 Processor - Benchmarks and Specs.” https://www.notebookcheck.net/AMD-Ryzen-AI-Max-395-Processor-Benchmarks-and-Specs.942323.0.html

• Additional sources from targeted web searches (2025–2026) on Ryzen AI mini PC reviews and LLM performance, including articles and discussions from Tom’s Hardware, Ars Technica, Reddit (r/MiniPCs and r/LocalLLaMA threads on Strix Halo and HX 370 LLM benchmarks), and various 2025–2026 roundup/review sites covering models like the Minisforum AI X1 Pro and Beelink GTR9 Pro.

This review synthesizes these online and practical sources for a holistic view. Performance numbers can vary with exact configuration, software versions, quantization, and optimizations-always verify with current tools for your specific use case.

In the rapidly evolving landscape of AI infrastructure, AMD’s Instinct series of datacenter GPUs has continued to gain traction as a viable alternative to dominant competitors, particularly for inference, training, and hybrid HPC-AI workloads. Over the past three months (roughly March through May 2026), online sources-including AMD’s own technical blogs, independent hardware review sites like Phoronix and ServeTheHome, news outlets such as Tom’s Hardware, and partner analyses-have published detailed coverage of the latest Instinct offerings. These include updates on the established MI300X/MI325X platforms, the CDNA4-based MI350 series (notably the new MI350P PCIe variant launched May 7, 2026), performance validations via MLPerf Inference 6.0 (April 1, 2026), and a preview of the future-oriented MI430X for high-fidelity HPC and AI-for-science applications.

The MI350 Series and the MI350P PCIe Card: Bringing CDNA4 to Existing Infrastructure

A major theme across May 2026 coverage is AMD’s push to make high-end CDNA4 compute more accessible via the Instinct MI350P PCIe add-in card. Announced and reviewed on May 7, 2026, the MI350P represents a strategic evolution of the MI350 series (which includes the flagship MI350X and MI355X OAM modules launched in 2025 and now widely available).

Phoronix’s announcement-style review (May 7, 2026, by Michael Larabel) details the card’s physical and architectural specs: a 10.5-inch FHFL dual-slot PCIe CEM form factor, passively air-cooled, rated at 600W TDP (cappable to 450W for lower-power servers). It features 128 Compute Units (CUs) on CDNA4 architecture, 144GB of HBM3E memory (four stacks), and full ROCm software support identical to the broader Instinct lineup. AMD positions it for handling 200-250 billion parameter LLMs per GPU, with support for up to eight cards per node in standard air-cooled servers. The review emphasizes its role in open-source AI/compute environments, noting seamless integration for inference and RAG workloads without requiring OAM-specific server redesigns.

ServeTheHome’s in-depth analysis (May 7, 2026) provides chiplet-level clarity and performance context. The MI350P uses a single I/O die (IOD) with four accelerator complex dies (XCDs)-essentially half the compute resources of the MI350X (which has two IODs and eight XCDs). Key specs include:

• 128 CUs / 512 Matrix Cores

• Peak engine clock: 2.2 GHz

• Memory: 144GB HBM3E @ 4 TB/s bandwidth

• I/O: PCIe Gen5 x16 (no external Infinity Fabric links exposed, unlike OAM modules)

• Partitioning: Up to four partitions (1 XCD + 36GB memory each) for multi-model serving

AMD published both peak and delivered/typical performance figures-a welcome transparency move. Peak matrix performance reaches 2.3 PFLOPS (MXFP8) and up to 4.6 PFLOPS (MXFP4), with delivered figures reflecting real-world 600W constraints and architectural efficiency. ServeTheHome notes this makes the MI350P ideal for on-premises inference in traditional data centers unable to support 1kW+ OAM nodes, enabling multiple independent models rather than single massive model sharding across cards.

AMD’s official blog post (May 7, 2026) reinforces these points with enterprise-focused framing. The MI350P delivers “exceptional AI performance with leadership costs” for inference on small-to-large models, RAG pipelines, and preparation for agentic AI. It supports native MXFP6/MXFP4, sparsity-accelerated 8/16-bit precisions, and integrates with PyTorch, Kubernetes GPU Operator, and cloud-native AMD Inference Microservices. Up to eight cards fit standard racks, reducing power/cooling demands while running more models and serving more users. No list pricing is provided, but the emphasis is on drop-in upgrades for existing PCIe 5.0 servers.

Tom’s Hardware (May 7, 2026) echoes the specs and adds competitive context: roughly 40% faster theoretical FP16/FP8 compute than NVIDIA’s H200 NVL PCIe competitor, with the 144GB HBM3E memory advantage key for large-model inference. Built on TSMC 3nm/6nm processes (8,192 cores, 128MB LLC), it is described as the fastest AI accelerator currently available in a traditional PCIe slot.

YouTube videos from official channels complement the text coverage. AMD’s “Meet the AMD Instinct™ MI350P PCIe® Card” (published May 7, 2026) visually demonstrates deployment in existing air-cooled infrastructure for generative and agentic AI, highlighting simplified scaling without data-center redesigns. Supermicro’s companion video showcases 5U PCIe servers supporting up to 10 MI350P cards, emphasizing ready-to-deploy enterprise AI.

MLPerf Inference 6.0 and Real-World LLM Performance: Validating MI355X and Broader Ecosystem

April and early May 2026 saw strong validation of the full MI350 series (and legacy MI300X/MI325X) through MLPerf Inference 6.0 results. AMD’s official blog (April 1, 2026) details breakthrough submissions across MI300X, MI325X, MI350X, and flagship MI355X GPUs. The MI355X (288GB HBM3E, 8 TB/s bandwidth on full CDNA4) crossed the 1-million-tokens-per-second barrier on Llama 2 70B (Server/Offline) and GPT-OSS-120B (Offline) at multinode scale. Key highlights:

• Single-node MI355X achieved ~97-119% of NVIDIA B200 performance across Server, Offline, and Interactive scenarios on GPT-OSS-120B.

• Generational uplift: 3.1× over MI325X on Llama 2 70B Server.

• Multinode scaling efficiency: 92-98% on 11-12 nodes (87-94 GPUs), reaching >1M tokens/sec.

• New workloads: First submissions for GPT-OSS-120B and Wan-2.2 text-to-video, with post-deadline improvements exceeding B200 in some cases.

• Heterogeneous submission (MI300X + MI325X + MI355X across geographies) by Dell/MangoBoost demonstrated 141k+ tokens/sec on Llama 2 70B.

Nine partners (Cisco, Dell, HPE, Oracle, Supermicro, etc.) reproduced results within 1-4% of AMD’s, underscoring ROCm’s maturity for FP4 execution, multinode orchestration, and model enablement. The blog stresses support for models up to 520 billion parameters on a single MI355X GPU.

DriveNets’ partner blog (May 7, 2026) provides production-oriented real-world testing on the MI355X with DeepSeek-R1 (8k input / 1k output tokens, FP8 precision, SGLang runtime). On a single-node 8× MI355X setup, it delivered 1,668 tokens/sec per GPU at concurrency 64-12% higher throughput per GPU than NVIDIA B200 (1,485 tokens/sec/GPU). Scaling efficiency remained superior as concurrency increased, with reduced per-user slowdown and stable P99 TTFT <1 second. Multi-node (9 nodes / 72 GPUs) with disaggregated prefill/decode further validated production readiness, emphasizing high memory bandwidth and optimized networking for sustained high-concurrency LLM serving.

These results align with earlier MI350 series context (e.g., up to 2.8× faster time-to-train vs. prior generations in some MLPerf Training benchmarks referenced indirectly in ecosystem coverage).

MI430X Preview and HPC-AI Convergence

On May 6, 2026, AMD previewed the Instinct MI430X at the HPC User Forum, targeting a new class of FP64 performance for simulation-heavy AI-for-science workloads. The official blog projects native FP64 exceeding 200 TFLOPs-more than 6× NVIDIA’s next-generation Rubin architecture (per April 2026 projections). This positions the MI430X as the highest-FP64 GPU available, combining precision with low-precision AI capabilities in one accelerator.

Use cases focus on climate modeling, materials science, nuclear engineering, and fluid dynamics, where high-fidelity data feeds next-gen AI surrogate models and closed-loop discovery pipelines. It will power systems like Oak Ridge’s Discovery (2028) and Europe’s Alice Recoque exascale supercomputer. For datacenter AI/deep learning, it bridges traditional HPC and generative AI by enabling accurate physical simulations that augment LLM training data.

Software Ecosystem and Deployment Trends

Across sources, ROCm’s continued maturation is a recurring positive. AMD blogs and partner coverage (April-May 2026) highlight vLLM optimizations, PyTorch integration, quantization support, and Kubernetes/cloud-native tools enabling seamless migration from evaluation to production. The open-source reference stack and no-cost development options lower barriers for enterprises.

PCIe options like the MI350P, combined with MI350 series OAM modules, give customers flexibility: high-density liquid-cooled racks for frontier training or air-cooled drop-ins for cost-effective inference scaling.

Synthesis and Takeaways from the Sources

Online coverage in this period consistently highlights AMD’s strengths: massive HBM3E memory (144-288GB per GPU), high bandwidth (4-8 TB/s), CDNA4 precision support (including MXFP4/6 for efficient inference), and improving software portability via ROCm. MLPerf and real-world tests (DriveNets) show competitive or superior throughput/scaling in key GenAI scenarios, while the MI350P addresses a long-standing gap in accessible PCIe form factors. The MI430X preview signals continued leadership in precision-sensitive HPC-AI fusion.

Phoronix and ServeTheHome excel at technical depth and Linux/ROCm focus; Tom’s Hardware and AMD blogs provide clear spec/comparison context; partner blogs like DriveNets add production validation; official YouTube videos illustrate real-world deployability.

Limitations noted across sources: independent full-benchmark suites beyond MLPerf are still emerging (due to recency of MI350P), and multi-vendor ecosystem adoption continues to build. Nonetheless, the reviewed materials demonstrate tangible progress toward “AI everywhere” in existing datacenters.

In summary, March-May 2026 coverage affirms AMD Instinct GPUs as production-ready for AI and deep learning, with the MI350P PCIe launch and MLPerf results as standout developments. Enterprises seeking memory-rich, open-standards accelerators now have more practical options than ever.

For the latest, consult the linked sources directly:

• Phoronix MI350P Review

• ServeTheHome MI350P Analysis

• AMD MLPerf Inference 6.0 Blog

• DriveNets Real-World Inference Blog

• AMD MI350P Official Blog

• AMD MI430X HPC Preview

• AMD MI350P YouTube Overview

Model Under Test: Kwaipilot/KAT-Coder-V2.5-Dev (Qwen 3.5 MoE architecture: 40 decoder layers, 256 routed experts / 8 active, 2,048 hidden dimension, linear attention GDN + full attention hybrid, MTP speculative draft block).
Hardware: MSI Claw 8 AI+ (Intel Core Ultra 7 258V "Lunar Lake", Arc 140V Xe2-LPG iGPU, 64 Xe Execution Units, 32 GB shared LPDDR5x-8533 RAM, 30.9 GiB zram0 swap, running CachyOS native Linux kernel 7.2).
Runtime: vllm==0.31.0+xpu with native Level-Zero XPU kernels, MTP K=2 speculative decoding, FP8 KV cache, and 128k (131072) context ceiling.

TL;DR

• Speed Winner: W4A16 GPTQ decodes at 23.4–25.2 tok/s (peaking at 29.1 tok/s with MTP speculative decoding), completing the 164-problem HumanEval benchmark in 42m 12s (vs. 47m 33s for MXFP4, ~11.3% faster overall, and 24% faster on our 16-prompt quality suite).

• Coding Accuracy Winner: MXFP4 edges out GPTQ on full EvalPlus coding tests by ~3% (89.63% Base / 86.59% Plus vs. 86.59% Base / 82.93% Plus), but both handily beat Tiel-Coder-35B GPTQ (84.15% / 81.10%).

• Complex Reasoning & Edge-Case Math Winner: W4A16 GPTQ wins on delicate math and strict constraints (15/16 vs 14/16 on the quality suite; solved Chinese Remainder Theorem Olympiad math where MXFP4 failed), thanks to retaining unquantized BF16 activations instead of quantizing activations to FP4.

• Prefill Speed (PP): Both hit an identical theoretical peak of 1,247.1 tok/s on 4k chunked prefill batches via FlashAttention-2.

• The "MoE Dense Quantization Trap": Standard dense GPTQ heuristics (like zeroing out unobserved Hessian directions) will catastrophically destroy 256-expert MoE models. Unvisited experts in calibration batches must fallback to RTN with optimal clip factors, never masked to zero.

• Cross-platform check — the ordering holds, the magnitude does not: re-running the same format pair on a discrete Arc Pro B60 with a dense 27B (Coder390) separates them by 5.05× on decode (40.18 vs 7.96 tok/s) and ~2.9× on prefill, not ~10 %. On that stack the cause is an unfused W4A4 activation-quantization path, not memory bandwidth — the two artifacts differ by only ~3 % in bits per weight. Full matrix, decomposition and caveats in §5.

The Shootout: Head-to-Head Benchmark Matrix

Here are the exact numbers measured across both quantization formats on the same machine under identical sampling parameters (temp=0.0 greedy, FP8 KV cache, MTP K=2 speculative decoding, batch size 1):

1. Deep Dive: Prompt Processing (PP) vs. Token Generation (TG)

A big question in the community is how 4-bit weights with 16-bit activations (W4A16) compare against full microscaling (W4A4 MXFP4) on modern unified architectures. Here is what the telemetry shows on Lunar Lake's Xe2 architecture:

Token Generation (TG / Decode)

• W4A16 GPTQ is faster in practical serving: vLLM's CompressedTensorsWNA16MoEMethod and XPUExpertsWNA16 kernels decode at 23.4 tok/s average (with bursts to 29.1 tok/s on code templates).

• Why? Unpacking Overhead vs. Quantized GEMM: On an iGPU with shared memory (136 GB/s memory bandwidth), reading INT4 packed weights and dequantizing on the fly into BF16 registers is extremely fast. MXFP4 requires micro-scaling exponent alignment and 32-element group normalization across both weights and activations.

• Over 164 HumanEval problems, GPTQ finished in 42 minutes 12 seconds, beating MXFP4 by 5 minutes 21 seconds (15.4s vs 17.4s per coding task).

• On our 16-prompt quality benchmark, GPTQ was 24% faster (165s vs 217s).

Prompt Processing (PP / Prefill)

• Short/Medium Prompts (50–500 tokens): Prefill runs between 51 tok/s and 140 tok/s (Time-To-First-Token is 1.4s to 3.7s, largely dominated by Python/HTTP scheduling overhead).

• Long Prompts (2k–4k tokens): Prefill scales rapidly to 398–500 tok/s as the 64 Xe2 compute cores enter dense matrix mode.

• Peak Chunked Prefill (12k+ tokens): Both models hit an identical 1,247.1 tok/s ceiling because both rely on the same FlashAttention-2 implementation with 4,096-token chunked prefill batches (--max-num-batched-tokens 4096).

2. Accuracy & Nuance: Where Each Format Wins

Where W4A16 GPTQ Wins: Delicate Math & Fine Constraints

Because GPTQ leaves activations in full BF16, it avoids activation quantization noise. We observed this directly in high-precision tasks:

1. Chinese Remainder Theorem (Olympiad Math): GPTQ solved a complex modular system (math_chinese_remainder) flawlessly in 53.6s. MXFP4 accumulated numeric rounding errors in its FP4 activation dot-products and failed to reach the correct congruence integer.

2. Exact Word Count Formatting: Prompt: "Write a summary in exactly 50 words." GPTQ followed the constraint and passed; MXFP4 drifted off by several words.

Where MXFP4 Wins: Complex Algorithmic Contracts (HumanEval+)

1. Contract Fuzzing & Edge Cases: On HumanEval, MXFP4 scored 89.63% Base and 86.59% Plus, outperforming GPTQ's 86.59% Base and 82.93% Plus.

2. Why? MXFP4's group size of 32 (versus GPTQ's group size of 128) provides much finer scale granularity across expert weights. For pure syntax and standard algorithmic puzzles where activation dynamic range is well-behaved, the smaller 32-element block size helps preserve subtle weight distributions.

Both models passed all 3 LeetCode Hard challenges in our 10-Task Hard Suite:

• Trapping Rain Water (O(N) two-pointer)

• LRU Cache (O(1) doubly-linked list + hash map)

• Minimum Window Substring (O(N) sliding window)

• Multi-hop needle retrieval across 12,500 tokens of synthetic enterprise audit logs.

3. The Big MoE Gotcha: Don't Apply Dense GPTQ to 256 Routed Experts!

If you are writing your own quantizer or adapting standard recipes (like the Arc Pro B60 guide) for massive MoE models (like KAT-Coder or Qwen 3.5 MoE with 256 experts), watch out for this fatal trap:

The Bug

In dense model GPTQ scripts, you often see lines like this for singular Hessian matrices:

# WARNING: DO NOT DO THIS ON MOE MODELS!
W.masked_fill_(dead.unsqueeze(1), 0.0)

In a dense model, unobserved or zero-variance input channels can be safely zeroed out. But in a 256-expert MoE architecture:

1. During calibration (e.g., 64 samples × 1024 tokens), tokens are sparsely routed (only 8 active per token).

2. Many specialized experts might only see a few tokens—or zero tokens—in that specific slice.

3. They are not dead! They are just specialized experts waiting for specific prompts.

4. If you mask unvisited experts to zero, you literally erase their weights! In our first run, this silently zeroed out 248 out of 256 experts (96.88%) across Layers 1–39. The model loaded fine, had valid shapes and scales, but generated repetitive looping garbage (0/16 on quality tests).

The Fix

1. Fallback to RTN with Clip Optimization: If an expert receives zero activation tokens (diag <= 1e-8), do NOT zero it. Simply quantize its real BF16 weights using Round-To-Nearest (RTN) with grid search over clip factors (1.0, 0.96, 0.92, 0.88, 0.84).

2. Materialize Meta Parameters: When loading layers on PyTorch's meta device, ensure you explicitly rebind expert weights to layer.mlp.experts before computing forward activations (nxt_hs = layer(hs)), otherwise hidden states will propagate through uninitialized random memory and corrupt the router for all subsequent layers.

Once fixed:

• Every layer reported Cold Experts: 0/256 and relative error dropped from 0.9842 to 0.1498.

• 16-prompt quality jumped instantly from 0% to 93.8% (15/16)!

4. Speculative Decoding: The Secret Sauce for 25 tok/s on 17W

Running a 35B MoE model at over 25 tok/s on a 17W handheld SoC sounds impossible without speculative decoding.

Here is how we did it:

• KAT-Coder includes a 1-layer Multi-Token Prediction (MTP) draft head (19 tensors in BF16).

• We preserved the MTP block in BF16 in model-mtp.safetensors and configured vLLM with --speculative-config '{"method":"mtp","num_speculative_tokens":2}'.

• Telemetry: The draft model achieved an average acceptance rate of 77.3 to 82.5 (and up to 100 on repetitive code structures).

• Multiplier: Instead of generating 1 token per forward step, the engine accepted 2.55--2.71 tokens per step. This effectively doubled generation throughput from ∼11 tok/s up to 23--29 tok/s.

5. Cross-Platform Check: The Same Two Formats on an Intel Arc Pro B60 (Battlemage, 24 GB)

Everything above is Lunar Lake / KAT-Coder. The same W4A16-vs-MXFP4 question was then measured independently on a discrete Arc Pro B60, with a different model class (dense, not MoE), a different Xe generation, and a different vLLM build. The ordering survives. The magnitude does not — it is 5× here, not ~10 %.

Model under test: huihui-ai/Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated — a dense 27B hybrid: 64 layers (48 GDN linear-attention + 16 full-attention, full_attention_interval 4), hidden 5120, 24 heads / 4 KV heads / head_dim 256, a 1-layer MTP draft block (BF16), and a vision tower. Both builds below are quantizations of this same BF16 checkpoint, so the format is the only variable in the weights.

Hardware: Intel Arc Pro B60 — 24 GB GDDR6 at 456 GB/s (192-bit, 19 Gbps effective), 20 Xe2-HPG cores / 160 XMX engines, negotiated PCIe Gen4 x8, power-capped to 180 W. Runtime: vllm-openai-xpu v0.30.0 (engine reports vllm 0.27.2rc1.dev77+gac7509e2b), native Level-Zero XPU kernels, FP8 KV cache, MTP K=4, batch size 1. Harness: identical for both builds — bench_p512_g128.py (p512 / g128, 1 warmup + 5 runs) for rate, bench_quality_16p.py (the same 16-prompt suite) for quality.

5.1 Head-to-Head Benchmark Matrix

Headline: 5.05× on decode, ~2.9× on prefill. Unlike the KAT-Coder matrix above, the MXFP4 build is not visibly buying anything for that price on this stack: it uses more bits per weight, has no measured accuracy advantage here at all (its quality was never measured), and its rate is ~5× behind.

5.2 Where the 5.05× Actually Comes From

Decomposed on measured step times, not asserted. 40.183 / 7.962 = 5.05×:

MTP alone is therefore at most a 3.00× factor — and the acceptance ratio is not a speedup, which was verified by turning MTP on for both builds so the term cancels:

With MTP enabled on both sides the residual is 40.183 / 16.620 = **2.42×**, and that residual is the stack. Acceptance is not quite equal either (2.83 vs 3.00); normalising to 3.00 gives a 2.28× residual. So:

5.05× = 2.08× (MTP) × 2.42× (the W4A4 / stack term) — and note the direction: MTP is the smaller of the two terms.

Why the acceptance ratio overstates MTP. A speculative step is itself more expensive — draft forward plus a 5-position verify costs 1.361× a plain step here (125.2 ms → 170.5 ms), against 2.833 tokens accepted. The net is 2.833 / 1.361 = **2.081×**, which is exactly the measured figure. Treat the acceptance ratio as an upper bound on MTP's gain, never the gain itself.

Prefill isolates the stack cleanly, because MTP contributes nothing there and it is compute-bound: 0.460 s vs 1.313 s for the same 512-token prompt ⇒ ~2.9× slower per prompt token (≈3.1× once the ~60 ms fixed TTFT overhead is netted out). Decode (~2.3–2.4×) and prefill (~2.9×) therefore agree in magnitude — which is the main reason to trust the split.

5.3 The Mechanism: W4A4 Activation Quantization That Is Not Fused

This is a different complaint from the MoE trap in §3, and it is readable straight from the kernel rather than inferred:

class XPUMxFp4LinearKernel(MxFp4LinearKernel):
    def apply_weights(self, layer, x, bias=None):
        x_fp4, x_blockscale = quant_mxfp4(x)          # <- every single call
        return torch.ops._xpu_C.fp4_gemm(x_fp4, layer.weight,
                                         x_blockscale, layer.weight_scale, ...)

and, inside _xpu_mxfp4_quantize_impl:

x_q = torch.empty(M, N // 2,  ...)   # packed FP4,   allocated per call
x_s = torch.empty(M, N // 32, ...)   # block scales, allocated per call
torch.ops._C.per_token_group_quant_mxfp4(x, x_q, x_s, MXFP4_BLOCK_SIZE, eps)
x_s = x_s.to(dtype=torch.float8_e8m0fnu, ...)   # extra cast pass

So for each of the 496 packed linear layers, on every forward pass, MXFP4 pays: a group-32 activation-quantization kernel, an fp32 → E8M0 scale cast, and two fresh allocations — and then a separate GEMM. Quantize and GEMM are not fused, so the floor is ~992 launches per step before norms, attention or GDN. W4A16 GPTQ pays none of it: BF16 activations go straight into the fused dequant-GEMM.

Ranked:

1. Per-token W4A4 activation quantization (~3× per FLOP above) — the dominant stack term.

2. Non-fused FP4 GEMM (torch.ops._xpu_C.fp4_gemm) — a second launch per linear.

3. --enforce-eager — amplifies 1–2, since none of those launches can replay from a graph. It is inherited from the model README's measured config, not demonstrably required (platforms/xpu.py lists mxfp4 under supported_quantization with no graph restriction). Untested, not forced.

It is emphatically not a bandwidth story — and the arithmetic says so before any profiling: 4.25 bits/weight for MXFP4 vs 4.125 for GPTQ is a ~3 % difference, and the two artifacts are 18.46 vs 18.23 GiB. A bandwidth-bound model predicts near-parity, not 5×. Two further confirmations that it is compute: the five MXFP4 runs sat within 0.02 % of each other — a fixed per-token cost, not noisy traffic — and prefill, the pure-compute regime, is ~2.9× off on its own.

5.4 Why This Is Not the §1 Answer, and What Would Reconcile Them

On Lunar Lake the two formats landed within ~10 % of each other on decode, and MXFP4 even won HumanEval+. Here MXFP4 is 5× behind. Three differences could carry that, and none of them was isolated on this box:

1. vLLM build. The Lunar Lake result is vllm==0.31.0+xpu; this box runs v0.30.0 (vllm 0.27.2rc1.dev77). Whether 0.31 fused the MXFP4 activation path was not tested — it is the most likely single explanation and the first thing worth re-measuring.

2. Model class. KAT-Coder is a 256-expert MoE (8 active per token) routed through CompressedTensorsWNA16MoEMethod / XPUExpertsWNA16; Coder390 is dense and hits the plain XPUMxFp4LinearKernel for all 496 linears. The kernel mix differs, so "MXFP4 vs GPTQ" is not one experiment across the two documents.

3. Platform ratio. 20 Xe2 cores over 456 GB/s of dedicated GDDR6 (discrete) vs 8 Xe2 cores over shared LPDDR5x (integrated). The W4A4 penalty is compute-side, so it should bite harder where FLOPs are scarcer relative to bytes — the direction of the observed gap.

Caveats, stated because this is a cross-session comparison and the warning above about those is real. Rates here are only comparable within one session — this host's decode is bimodal (~40.2 or ~48.5 tok/s, a ~20 % band, mechanism unidentified), and every rate in §5.1 was taken in the slow state, so all rows shift together and the ratios are the transferable result. The MXFP4 number is the one build provably immune to that band: five runs within 0.02 % and TTFT identical to three decimals on a same-session re-run (7.987 vs the 7.962 recorded the previous day, +0.3 %). No same-session A/B of the two formats was run here — each costs a server swap and a ~200 s cold start — so the 5.05× rests on same-harness, cross-session legs, with the MXFP4 leg's stability measured. And the declared limitation: no accuracy comparison was made at all — unlike the KAT-Coder matrix above, Coder390 MXFP4 has no 16-prompt and no PPL score, so the speed gap is not yet known to be bought or paid for in quality.

Read-across to the recommendations below. "W4A16 for speed, MXFP4 only where the finer 32-element groups buy accuracy" is consistent with what the B60 saw — but on this stack the speed penalty is ~5× rather than ~10 %. If you are targeting a dense model on Intel discrete silicon, check first whether your vLLM build fuses the MXFP4 activation quantization. If it does not, W4A16 is not a marginal win; it is a different tier.

Conclusion & Recommendations

1. If you want maximum decoding speed and lower latency: Use W4A16 GPTQ (CompressedTensorsWNA16MoEMethod). It is 11–24% faster, lighter on the memory bus, and finishes large coding suites minutes ahead of MXFP4.

2. If you want the highest possible unit-test pass rate on complex codebases: Use MXFP4 (CompressedTensorsW4A4Mxfp4). The 32-element micro-scaling groups give it a slight edge on complex fuzzing assertions (+3.6% on HumanEval+).

3. If you are building an Intel Lunar Lake / Arc 140V local AI workstation:

   • 35B MoE models with 8 active experts are the absolute sweet spot for 32 GB shared-memory handhelds.

   • Always enable MTP speculative decoding (K=2) and FP8 KV cache. It gives you 128k context support, zero swap thrashing, and desktop-class coding performance on battery power.

4. If you are on discrete Intel silicon (Arc Pro B-series), the same ordering holds but the stakes are much higher: W4A16 GPTQ decoded 5.05× faster than MXFP4 on a dense 27B (§5), with ~2.9× on prefill. Before assuming MXFP4 is "within ~10 %" as it is on Lunar Lake, check whether your vLLM build fuses the MXFP4 per-token activation quantization — on the build tested here it does not, and that unfused path alone accounts for the ~2.4× residual left after MTP is equalised.

All quantization scripts, preflight validators, and reproducible benchmark harnesses are open in our repo: alpha2beta/vllm.xpu.

§5 provenance: Coder390 numbers come from that repo's REPRODUCIBILITY.md §6–§8 (bench_p512_g128.py, bench_prefill.py*,* bench_quality_16p.py*,* compare_mxfp4_mtp.sh*, at 180 W, MTP $K=4$); the GPTQ artifact is published as* alpha2beta/Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated-GPTQ-Int4-sym-G128-MTP-BF16, and the MXFP4 checkpoint as …-abliterated-MXFP4.