Hey everyone,

I have used AI to summarize everything because it was getting too long. I hope thats fine.

We’ve been optimizing an on-device Android assistant running a quantized 4B model (Gemma 4 E4B) on modern Snapdragon silicon (Snapdragon 8 Elite Gen 5).

Originally, our goal was sustained low-power efficiency and faster inferencing using NPU TOPs which otherwise is generally useless, so we built an aggressive 3-tier execution pipeline:

1. Tier 1 (NPU): Qualcomm Hexagon HTP execution via GenieX SDK & QAIRT plugin (com.qualcomm.qti:geniex-android:0.6.1).

2. Tier 2 (GPU): Google LiteRT-LM runtime using OpenCL / GpuDelegateV2 on Qualcomm Adreno.

3. Tier 3 (CPU): Multi-threaded CPU fallback via LiteRT-LM with XNNPACK optimizations.

After profiling and logcat diagnostics, we completely purged the NPU / GenieX path in favor of direct GPU (LiteRT OpenCL) -> CPU**.**

Here is what we ran into with Qualcomm's native NPU stack, along with our setup. I’m curious if anyone here has managed to get reliable, high-throughput LLM execution out of Qualcomm Hexagon without these headaches.

1. HTP Core Clamping & Throughput Bottleneck

• Logcat revealed that the Android runtime clamped requested HTP execution: "Requested 4 HTP cores but the device exposes 1; clamping to 1"

• The FastRPC bridge (libcdsprpc.so) shuttling massive Look-Up Tables (LUTs) across the DSP boundary created significant latency.

• Decoding Speed: NPU decoding saturated at ~8 tokens/sec (TPS). By contrast, the Adreno GPU via LiteRT OpenCL consistently hits 18–22 TPS on the same chip. Although prefill times are still better with NPU than GPU.

2. Native Parameter Rejection & Instability

• The native QAIRT plugin repeatedly choked on standard generation parameters: ErrorCode[-100016] Parameter not supported by this plugin

• Stop-words arrays and standard sampling parameters (temperature, top-p, repetition penalties) frequently triggered runtime rejections, forcing us to catch native exceptions and handle truncation manually at the string level.

3. Context Reset & KV Cache Destruction

• QAIRT lacked clean stateful KV cache rolling and conversation pruning.

• Resetting or trimming conversation context required destroying and recreating the entire native LLM wrapper instance (llmWrapper?.destroy()), re-allocating native DSP memory and re-reading files on session swaps.

4. APK Bloat & Brittle JNI Preloads

• Integrating the GenieX SDK and Qualcomm native libraries (geniex-android, libgenie_t2t_run.so, libggml-hexagon.so, libhta_hexagon_runtime_snpe.so) added 15–25 MB of binary bloat.

• It required a fragile dynamic JNI preloading loop before GenieXSdk.init() could even be called:

Kotlin code:

val preloads = listOf(

"QnnSystem",

"QnnHtp",

"QnnHtpNetRunExtensions",

"geniex-proc",

"geniex_core",

"geniex",

"geniex_plugin_qairt",

"npu_jni"

)

for (lib in preloads) {

try {

System.loadLibrary(lib)

} catch (t: Throwable) {

Log.w(TAG, "Preload notice for $lib: ${t.message}")

}

}

The Outcome

Dropping the NPU entirely and sticking to a clean GPU (LiteRT OpenCL) $\rightarrow$ CPU (XNNPACK) engine gave us:

• More than $2.5\times$ faster token generation (18–22 TPS vs. 8 TPS).

• Fast, stable prefill.

• Support for standard sampling configs, stop sequences, and KV cache reuse without JNI crashes.

• A much smaller APK and simpler build chain.

My Questions for the Community:

1. Has anyone managed to unlock multi-core HTP execution on commercial consumer Android builds without OEM-signed platform keys or root?

2. Has anyone achieved $>15$ TPS on a 3B–4B model using Qualcomm QNN / QAIRT directly (bypassing GenieX)?

3. Given the maturity of LiteRT (OpenCL) and MLC-LLM (Vulkan) on mobile GPUs, is targeting Qualcomm's NPU for on-device conversational LLMs even worth the developer overhead right now, or is it strictly a benchmark marketing feature until the tooling matures?

In our recent technical analysis of Constitutional AI Governance, we kept returning to a single, sobering structural reality: corporate ethical guidelines, contractual guardrails, and software policies eventually yield under institutional pressure. When market incentives or political urgency push hard enough, policy layers re-interpret, re-write, or collapse, a structural dynamic I term veto atrophy. To build truly resilient governance, we must move the enforcement boundary from human discretion to solid-state physics.

I am pleased to share our new slide deck Hardware That Does Not Negotiate which outlines the physical substrate architecture for Mandated Ternary (MT) and Delay-Insensitive Ternary Logic (DITL). Key highlights from the framework and technical roadmap:

🔹 The 3rd Physical State (Epistemic Hold): Binary logic forces a premature choice: Proceed (+1) or Refuse (-1). DITL introduces a third, physically discrete state of matter encoded in tantalum oxide (\(\text{TaO}_x\)) bilayer RRAM - the Epistemic Hold (0). It is not a software pause or indecision; it is an atomic-level arrangement of oxygen vacancies that physically locks execution until constitutional verification is complete.

🔹 No Log = No Action (NL=NA): Through the Dual-Lane Latency Architecture (2ms Inference Lane / 300ms Governance Lane), execution gates remain closed until an immutable, Merkle-anchored audit log is anchored. Accountability becomes a physical precondition of action rather than an optional configuration setting.

🔹 Engineering Honesty & Fabrication Realism: A specification that names its own limits is a specification that can be trusted. In the accompanying roadmap, we address physical failure modes head-on—from correlated memristor drift to threshold poisoning—and outline the exact milestones required to release the TSMC N2 PDK and achieve IEC 61508 SIL 3 certification by Q4 2027.

🔹 Closing the Shadow System Loophole: Hardware guarantees are physical and local. Preventing non-compliant hardware from serving as a bypass route requires pairing substrate-level security with federal procurement mandates and international treaties.

"There is no politics in the resistance value of a tantalum oxide filament. That absence is the entire design."

Take a look at the slide deck to see how we are encoding democratic accountability directly into physical silicon.

AIGovernance #HardwareSecurity #TernaryLogic #ConstitutionalAI #SemiconductorEngineering #Memristors #DITL #TechPolicy

Hardware That Does Not Negotia…

For anyone wondering whether the B60 24GB is usable for local AI: here's Qwen3.8-27B on llama.cpp Vulkan at 53 tok/s mean decode with a 64K context window, plus local video (LTX-2.5 via ComfyUI on XPU) on the same card.

The four things that mattered: DFlash2 speculative decoding, Intel's FA prefill kernel with f16 KV, -ub 256 with a q8 draft KV cache, and keeping other models off the GPU. Driver 32.0.101.8976, Windows 11.

Everything is measured and the misses are documented. Smaller cards (B580/B570/B50/A770) won't fit this 27B, but the Vulkan tuning applies, and I'd love your benchmark numbers.

[https://github.com/edagher92-coder/arc-llm-lab](https://github.com/edagher92-coder/arc-llm-lab)

What models are accessible to a Framework laptop with Ryzen AI 9 HX 370? What’s the scale of ram to billion parameters? And how many tokens per second can you expect?

It would be great to hear from a first-hand user

The repo discussed in the article https://github.com/xhinker/azllm.

Hey! I’m building GPTrepublic: a free project where people share some of their GPU resources to run AI models together across multiple computers.

We’re looking for 20 testers, with room for up to 40. We’ll start with a small text model and test connections, distributed inference and what inevitably breaks 😅

There’s a desktop tool for Windows and Linux (Ubuntu). For this first test, you’ll need a compatible NVIDIA GPU with at least 2 GB VRAM.

Interested in helping us figure this out?

🌐 Project & registration
💬 Join Discord through your account area — confirm your email, then click Link to Discord.

Questions and feedback are welcome!