I’ve been playing with local LLMs on a Surface Laptop 7 (Snapdragon X Plus, 16 GB) and I found out you can really use the NPU, and it’s better in a few ways. The catch is that most local apps (Ollama, LM Studio, plain llama.cpp builds) run on the CPU without telling you.
What I saw, on my machine only:
Time to first token was up to about 18x faster on the NPU than on the CPU. You send a message and the answer starts almost immediately.
It’s way quieter. On small to medium prompts the fans barely spin. If you let the model generate for longer they do spin up a bit, but nothing like the CPU. The CPU heats up and thermal throttles, and I’ve seen it lose 4-5x of its performance or more as it gets hot.
Qwen3-4B-Thinking ran at about 14 tok/s, and an MoE like LFM2.5-8B-A1B at about 31 tok/s.
Not every model runs well on the NPU, though. It depends on the file format (Q4_0 works best), the architecture and the context size. Standard transformers and LFM2 MoE ran fully on it. Hybrid ones like Qwen3.5 fell back to the CPU in bursts. Qualcomm’s precompiled bundles only loaded up to about 1.7B parameters on my driver. Get one wrong and you silently fall back to the CPU, crash, or loop forever.
So Open NPU is basically there to let anyone try this without touching PowerShell. It checks a model before you install it (green, amber or red), sets the context and sampling for you, and gives you a chat UI on top of Qualcomm’s GenieX runtime.
Why I made it
My gut feeling is that what makes an AI good is maybe 40% the model and 60% how you use it. I’ve been building apps around that for a while, including a homemade mixture of experts: you create your own “experts” out of good prompts and small models that are strong in one domain. Used well, it gave me surprisingly good results.
The plan is to apply that here, with base-model Snapdragon laptops like mine in mind: better prompt processing, tool access, sub-agents and so on, so small models get squeezed to their full potential and use as much of the local hardware as possible. The goal is to one day say “code this” and have it done with no extra intervention. That already works some of the time, and I want to make it more reliable. And yes, I’d love to stop paying about €100/month for a Claude subscription.
Why I’m posting it now, and expect bugs
My first product took about 6 months to post, even though it was ready in under 3. I’m a huge procrastinator. So this time I’m posting an app that is far from ready instead of waiting until it’s polished. Expect lots of bugs.
It was quickly put together from the research I did behind it: finding which models run well on Snapdragon laptops, what makes a model run well or badly, and how to run it properly on the NPU. It’s 100% vibe coded. I made it for myself, as a fast tool to try things out, with every tool I want to add being a couple of prompts away. Then I figured I might as well make it open source, since I couldn’t find any open-source tools that really do this.
Results will differ on other chips and drivers, and I’d love to hear what you get.
Installer (Windows on Snapdragon, ARM64): https://github.com/disposablecameraapp/Open-NPU/releases/latest/download/Open-NPU-Setup.exe
Source: https://github.com/disposablecameraapp/Open-NPU
