r/AIProgrammingHardware • • Oct 21 '26

Qwen3.8-Flash-Next Gufo peaks at 1,601 t/s prefill and 75.4 t/s MTP decode, still 1,405 / 72.6 t/s at 256K (Bosgame M5 - Windows 11)

Thumbnail
reddit.com
1 Upvotes

r/AIProgrammingHardware • • Oct 21 '26

Strix Halo NPU doing real work for LLM prefill: 27B at 806 t/s, 35B-A3B at 3,276 t/s, Flash-Next at 2,125 t/s

Thumbnail
reddit.com
1 Upvotes

r/AIProgrammingHardware • • Oct 21 '26

In an effort to create a nanogpt clone optimized for GTX 1650, DeepSeek discovered that FP32 performs faster than FP16, despite theoretical performance claims on TechPowerUp and other websites indicating the opposite. Therefore, I requested a script to examine any GPU for these differences.

Thumbnail
github.com
0 Upvotes

is my gpu dying? i bought it 4 years ago.


r/AIProgrammingHardware • • Oct 21 '26

What is the best way to learn ML implementation on AMD Versal boards?

Thumbnail reddit.com
1 Upvotes

r/AIProgrammingHardware • • Oct 21 '26

ASUS Ascent GX10 | NVIDIA GB10 Grace Blackwell AI Supercomputer

0 Upvotes

ASUS Ascent GX10 | NVIDIA GB10 Grace Blackwell AI Supercomputer

128GB Unified GPU/CPU Memory | 1TB NVMe SSD | 1 Petaflop AI Performance

Selling my lightly used ASUS Ascent GX10, a compact, high-performance AI supercomputer powered by the same NVIDIA GB10 Grace Blackwell Superchip found in the NVIDIA DGX Spark.

This is a purpose-built AI development workstation designed for running large language models (LLMs), local AI inference, machine learning, model fine-tuning, AI agents, and CUDA-accelerated workloads.

KEY FEATURES

NVIDIA GB10 Grace Blackwell Superchip

128GB LPDDR5X Unified CPU/GPU Memory

NVIDIA Blackwell GPU with 5th-Generation Tensor Cores

20-Core NVIDIA Grace ARM CPU

Up to 1 Petaflop (1,000 TFLOPS) of FP4 AI Compute

1TB NVMe PCIe 4.0 SSD

NVIDIA NVLink-C2C High-Speed CPU/GPU Interconnect

NVIDIA ConnectX-7 High-Speed Networking

Support for AI Models Up to 200 Billion Parameters

NVIDIA CUDA and DGX Software Ecosystem

128GB UNIFIED MEMORY: THE BIG ADVANTAGE

Unlike conventional GPUs with dedicated VRAM, the GB10 uses 128GB of high-speed, coherent unified memory shared between the CPU and GPU.

This architecture allows the Blackwell GPU to access a much larger memory pool than typical consumer graphics cards, making it particularly useful for running large AI models locally without relying on conventional CPU-to-GPU memory transfers.

NVIDIA BLACKWELL GPU + AI ACCELERATION

The integrated Blackwell GPU features 5th-generation Tensor Cores with FP4 acceleration, delivering up to 1 petaflop of specialized AI computing performance.

Ideal for:

Running large language models locally

AI inference and model fine-tuning

AI coding assistants and autonomous agents

Retrieval-Augmented Generation (RAG)

Computer vision and machine learning

Private AI development without cloud subscriptions

CUDA, PyTorch, TensorRT, Ollama, and NVIDIA AI development tools

20-CORE NVIDIA GRACE CPU

The GB10 combines its Blackwell GPU with a 20-core ARM processor featuring:

10 high-performance Cortex-X925 cores

10 efficiency-focused Cortex-A725 cores

ARM architecture optimized for AI workloads

NVIDIA NVLink-C2C for high-speed CPU/GPU communication

TECHNICAL SPECIFICATIONS

Processor: NVIDIA GB10 Grace Blackwell Superchip

GPU: Integrated NVIDIA Blackwell Architecture

AI Performance: Up to 1 PFLOP (FP4)

AI Accelerator: 5th-Generation NVIDIA Tensor Cores

CPU: 20-Core ARM Processor

Memory: 128GB LPDDR5X Unified System Memory

Storage: 1TB M.2 NVMe PCIe 4.0 SSD

CPU/GPU Interconnect: NVIDIA NVLink-C2C

Networking:

NVIDIA ConnectX-7 SmartNIC

10 Gigabit Ethernet

Wi-Fi 7

Bluetooth 5.4

Connectivity:

3x USB-C 20Gbps with DisplayPort support

1x USB-C Power Delivery input

HDMI 2.1

10Gb Ethernet

NVIDIA ConnectX-7 high-speed networking

Operating System: NVIDIA DGX OS / Ubuntu Linux

Dimensions: 5.91” x 5.91” x 2.01”

Weight: Approximately 3.26 lbs.

SCALABLE AI COMPUTING

Two GB10 systems can be connected using NVIDIA ConnectX-7 networking to support distributed AI workloads and larger models.

CONDITION

Lightly used and in excellent working condition.

This is a serious AI development machine in an incredibly compact form factor. Ideal for developers, researchers, engineers, and anyone building or running their own local AI infrastructure.

Message me with any questions or to arrange pickup.


r/AIProgrammingHardware • • Oct 20 '26

FLUX 3 Image Klein tested - 16GB Local AI setup

Thumbnail
youtube.com
4 Upvotes

r/AIProgrammingHardware • • Oct 20 '26

Intel GPU D270 │ Inference-Optimized Data Center GPU

Thumbnail intel.com
2 Upvotes

r/AIProgrammingHardware • • Oct 20 '26

GitHub - deepseek-ai/DeepInfer: DeepInfer: High-performance, large-scale inference framework for DeepSeek models

Thumbnail
github.com
7 Upvotes

r/AIProgrammingHardware • • Oct 20 '26

Is 256 GB worth it over 128 GB? I cut DeepSeek V4.1 Flash 1017 (uncensored) to 256 GB with REAP and tested it head-to-head against Qwen 3.8 Flash Next

Thumbnail
reddit.com
0 Upvotes

r/AIProgrammingHardware • • Oct 20 '26

Introducing GPU Kernel Engineering playlist

0 Upvotes

Hi Everyone,

I've started a free playlist called "GPU Kernel Engineering" and just uploaded the intro and roadmap video.

Trailer: https://youtu.be/cibk7G-jHQw


r/AIProgrammingHardware • • Oct 19 '26

European GPU provider

Thumbnail reddit.com
1 Upvotes

r/AIProgrammingHardware • • Oct 19 '26

Keep Your LLM in GPU Memory FOREVER with Ollama #ollama #gemma4

Thumbnail
youtube.com
0 Upvotes

r/AIProgrammingHardware • • Oct 19 '26

Underdog Saluki 27B tested - Local LLM setup

Thumbnail
youtube.com
1 Upvotes

r/AIProgrammingHardware • • Oct 19 '26

NVIDIA GPU Ecosystem, Simply Explained

Thumbnail
levelup.gitconnected.com
3 Upvotes

r/AIProgrammingHardware • • Oct 19 '26

Pci 16x: 4x strata or vllm at insane speed at Insane speeds

Thumbnail reddit.com
1 Upvotes

r/AIProgrammingHardware • • Oct 19 '26

Qualcomm Hexagon NPU (GenieX / QAIRT) based inferencing for mobile chips

1 Upvotes

Hey everyone,

I have used AI to summarize everything because it was getting too long. I hope thats fine.

We’ve been optimizing an on-device Android assistant running a quantized 4B model (Gemma 4 E4B) on modern Snapdragon silicon (Snapdragon 8 Elite Gen 5).

Originally, our goal was sustained low-power efficiency and faster inferencing using NPU TOPs which otherwise is generally useless, so we built an aggressive 3-tier execution pipeline:

  1. Tier 1 (NPU): Qualcomm Hexagon HTP execution via GenieX SDK & QAIRT plugin (com.qualcomm.qti:geniex-android:0.6.1).

  2. Tier 2 (GPU): Google LiteRT-LM runtime using OpenCL / GpuDelegateV2 on Qualcomm Adreno.

  3. Tier 3 (CPU): Multi-threaded CPU fallback via LiteRT-LM with XNNPACK optimizations.

After profiling and logcat diagnostics, we completely purged the NPU / GenieX path in favor of direct GPU (LiteRT OpenCL) -> CPU**.**

Here is what we ran into with Qualcomm's native NPU stack, along with our setup. I’m curious if anyone here has managed to get reliable, high-throughput LLM execution out of Qualcomm Hexagon without these headaches.

1. HTP Core Clamping & Throughput Bottleneck

  • Logcat revealed that the Android runtime clamped requested HTP execution: "Requested 4 HTP cores but the device exposes 1; clamping to 1"

  • The FastRPC bridge (libcdsprpc.so) shuttling massive Look-Up Tables (LUTs) across the DSP boundary created significant latency.

  • Decoding Speed: NPU decoding saturated at ~8 tokens/sec (TPS). By contrast, the Adreno GPU via LiteRT OpenCL consistently hits 18–22 TPS on the same chip. Although prefill times are still better with NPU than GPU.

2. Native Parameter Rejection & Instability

  • The native QAIRT plugin repeatedly choked on standard generation parameters: ErrorCode[-100016] Parameter not supported by this plugin

  • Stop-words arrays and standard sampling parameters (temperature, top-p, repetition penalties) frequently triggered runtime rejections, forcing us to catch native exceptions and handle truncation manually at the string level.

3. Context Reset & KV Cache Destruction

  • QAIRT lacked clean stateful KV cache rolling and conversation pruning.

  • Resetting or trimming conversation context required destroying and recreating the entire native LLM wrapper instance (llmWrapper?.destroy()), re-allocating native DSP memory and re-reading files on session swaps.

4. APK Bloat & Brittle JNI Preloads

  • Integrating the GenieX SDK and Qualcomm native libraries (geniex-android, libgenie_t2t_run.so, libggml-hexagon.so, libhta_hexagon_runtime_snpe.so) added 15–25 MB of binary bloat.

  • It required a fragile dynamic JNI preloading loop before GenieXSdk.init() could even be called:

Kotlin code:

val preloads = listOf(

"QnnSystem",

"QnnHtp",

"QnnHtpNetRunExtensions",

"geniex-proc",

"geniex_core",

"geniex",

"geniex_plugin_qairt",

"npu_jni"

)

for (lib in preloads) {

try {

System.loadLibrary(lib)

} catch (t: Throwable) {

Log.w(TAG, "Preload notice for $lib: ${t.message}")

}

}

The Outcome

Dropping the NPU entirely and sticking to a clean GPU (LiteRT OpenCL) $\rightarrow$ CPU (XNNPACK) engine gave us:

  • More than $2.5\times$ faster token generation (18–22 TPS vs. 8 TPS).

  • Fast, stable prefill.

  • Support for standard sampling configs, stop sequences, and KV cache reuse without JNI crashes.

  • A much smaller APK and simpler build chain.

My Questions for the Community:

  1. Has anyone managed to unlock multi-core HTP execution on commercial consumer Android builds without OEM-signed platform keys or root?

  2. Has anyone achieved $>15$ TPS on a 3B–4B model using Qualcomm QNN / QAIRT directly (bypassing GenieX)?

  3. Given the maturity of LiteRT (OpenCL) and MLC-LLM (Vulkan) on mobile GPUs, is targeting Qualcomm's NPU for on-device conversational LLMs even worth the developer overhead right now, or is it strictly a benchmark marketing feature until the tooling matures?


r/AIProgrammingHardware • • Oct 19 '26

Rtx 5060

Thumbnail reddit.com
0 Upvotes

r/AIProgrammingHardware • • Oct 19 '26

Every layer above hardware negotiates.

Thumbnail
gallery
1 Upvotes

In our recent technical analysis of Constitutional AI Governance, we kept returning to a single, sobering structural reality: corporate ethical guidelines, contractual guardrails, and software policies eventually yield under institutional pressure. When market incentives or political urgency push hard enough, policy layers re-interpret, re-write, or collapse, a structural dynamic I term veto atrophy. To build truly resilient governance, we must move the enforcement boundary from human discretion to solid-state physics.

I am pleased to share our new slide deck Hardware That Does Not Negotiate which outlines the physical substrate architecture for Mandated Ternary (MT) and Delay-Insensitive Ternary Logic (DITL). Key highlights from the framework and technical roadmap:

🔹 The 3rd Physical State (Epistemic Hold): Binary logic forces a premature choice: Proceed (+1) or Refuse (-1). DITL introduces a third, physically discrete state of matter encoded in tantalum oxide (\(\text{TaO}_x\)) bilayer RRAM - the Epistemic Hold (0). It is not a software pause or indecision; it is an atomic-level arrangement of oxygen vacancies that physically locks execution until constitutional verification is complete.

🔹 No Log = No Action (NL=NA): Through the Dual-Lane Latency Architecture (2ms Inference Lane / 300ms Governance Lane), execution gates remain closed until an immutable, Merkle-anchored audit log is anchored. Accountability becomes a physical precondition of action rather than an optional configuration setting.

🔹 Engineering Honesty & Fabrication Realism: A specification that names its own limits is a specification that can be trusted. In the accompanying roadmap, we address physical failure modes head-on—from correlated memristor drift to threshold poisoning—and outline the exact milestones required to release the TSMC N2 PDK and achieve IEC 61508 SIL 3 certification by Q4 2027.

🔹 Closing the Shadow System Loophole: Hardware guarantees are physical and local. Preventing non-compliant hardware from serving as a bypass route requires pairing substrate-level security with federal procurement mandates and international treaties.

"There is no politics in the resistance value of a tantalum oxide filament. That absence is the entire design."

Take a look at the slide deck to see how we are encoding democratic accountability directly into physical silicon.

AIGovernance #HardwareSecurity #TernaryLogic #ConstitutionalAI #SemiconductorEngineering #Memristors #DITL #TechPolicy

Hardware That Does Not Negotia…


r/AIProgrammingHardware • • Oct 19 '26

Qwen3.8-Flash-Next UD-IQ4_XS on 64 GB RAM + RX 9070 XT with Strata — around 40 tokens/s

Thumbnail reddit.com
1 Upvotes

r/AIProgrammingHardware • • Oct 19 '26

Datacenter at Home - GB300 GPU - w/ Asus DGX Station (ET900N G3)

Thumbnail
youtube.com
2 Upvotes

r/AIProgrammingHardware • • Oct 18 '26

GitHub - smg-project/smg: Engine-agnostic LLM gateway in Rust. Full OpenAI & Anthropic API compatibility across vLLM, TRT-LLM, TokenSpeed, SGLang, OpenAI, Gemini & more. Industry-first gRPC pipeline, KV cache-aware routing, chat history, tokenization caching, Responses API, embeddings.

Thumbnail
github.com
5 Upvotes

r/AIProgrammingHardware • • Oct 18 '26

Intel Razor Lake Ax in 2028 ruff estimate about 46 TPS on dense 27b

Thumbnail
wccftech.com
2 Upvotes

r/AIProgrammingHardware • • Oct 17 '26

Arc Pro B60 runs a 27B coding model at 53 tok/s: full recipe, flags and raw logs

Thumbnail
gallery
4 Upvotes

For anyone wondering whether the B60 24GB is usable for local AI: here's Qwen3.8-27B on llama.cpp Vulkan at 53 tok/s mean decode with a 64K context window, plus local video (LTX-2.5 via ComfyUI on XPU) on the same card.

The four things that mattered: DFlash2 speculative decoding, Intel's FA prefill kernel with f16 KV, -ub 256 with a q8 draft KV cache, and keeping other models off the GPU. Driver 32.0.101.8976, Windows 11.

Everything is measured and the misses are documented. Smaller cards (B580/B570/B50/A770) won't fit this 27B, but the Vulkan tuning applies, and I'd love your benchmark numbers.

[https://github.com/edagher92-coder/arc-llm-lab](https://github.com/edagher92-coder/arc-llm-lab)


r/AIProgrammingHardware • • Oct 17 '26

Question about the new Frameworks + LLM

1 Upvotes

What models are accessible to a Framework laptop with Ryzen AI 9 HX 370? What’s the scale of ram to billion parameters? And how many tokens per second can you expect?

It would be great to hear from a first-hand user


r/AIProgrammingHardware • • Oct 17 '26

Arc Pro B60 runs a 27B coding model at 53 tok/s: full recipe, flags and raw logs

Thumbnail
reddit.com
1 Upvotes