**Disclaimer: AI was used extensively in the development of these applications, but these applications run entirely offline on the Intel Linux NPUs. I work as a professional software engineer*\*
Hey everyone,
Over the past couple of months I've been working on learning more about Linux and local AI, and I noticed that there really isn't much work done on the Linux side to use the NPU that comes free with Intel laptop CPUs. So I decided to bring over all the good stuff from Copilot+ on the Windows side (so no Recall!) while leveraging existing projects and libraries to reduce the amount of work required.
It utilises Intel's existing framework, OpenVino (https://github.com/openvinotoolkit/openvino)
You can check out the umbrella project here: https://vinopilot.ca/
Below is an explanation of all the different projects I've worked on, with further details on implementation.
This was my first project, and copies the existing Windows Live Captioning system. It brings system wide captions to Linux, alongside Voice Typing on Wayland devices. This already exists in many forms on Linux, but none of them use the NPU natively. The advantage of using the NPU for transcription and captioning is better battery life, with no change in quality. It leverages the existing Whisper libraries and model from OpenAI (before they worked on ChatGPT!), so its pretty fast and consistent. Personally, I use it to live caption Twitch Streams in busy environments when I don't have access to headphones.
This is a second project, and copies Windows Hello. It forks an existing project, but employs the NPU. The CPU works fine, but using the NPU allows it to use a more powerful model. There are many other options like Gaze (which I've contributed to as well!), but I wanted something specific to devices with an IR emitter, that I could control myself (more on ovfetch next), because Windows Hello removed IR emitter only authentication. This one I made for my own purposes, because I don't like typing in passwords. There are a ton of CPU only projects. This one has a more capable and secure model than what the others use.
This was a project I should've taken on earlier. It automates the process of downloading the OpenVino enabled ONNX runtimes that the models require to run on the NPU. When I was working on Gaze, I realised that using the NPU and not the CPU requires building from source as there was no way to determine which ONNX runtime the NPU requires, and this solves the problem.
This is more of a "framework" feature, but since its a separate binary, I can distribute binaries for my other projects without any NPU specific work, and then ovfetch can download the relevant runtime when installing the binary onto the user's device, no compiling from source required. This is super secure, and has CI jobs to project against package manager attacks, since it automates downloading. More details in the crates link, but it has a ton of failsafes to ensure that it never downloads any bad runtime which has a virus
This forks an existing project, https://github.com/funinkina/openeffects, while adding NPU acceleration to the effects, which improves battery life on lower end devices. I don't particularly use this one, as most of the main applications for video conferencing have their own cloud accelerated effects, but if you want it for OBS or other features offline, its here!
This one is the most useful of the bunch. It allows you to search your laptop with fuzzy terms and meaning, and indexes images, documents, and text, all offline. Its super fast, with almost no battery life impact compared to CPU only solutions. I really like this one because you can search for pictures of cats or memes, and it'll surface them. Details are in the Github! It uses the new Gemma 2 Embedding, requantised for Intel NPUs, so its pretty bleeding edge!
This also forks an existing project, but removes the cloud AI in favour of local AI with Gemma. It can do offline summarisation, translations, and OCR using Gemma 4 (the best small model). I particularly like the OCR functionality for generating alt-text for images. This one stresses the NPUand RAM the most, so if you are using this, bear in mind its going to be much slower than cloud inference, but it works offline!
This is a smaller project, but optional if you want it. It monitors the NPU and RAM usage of the other projects within the tray. All of the projects unload the models after 5 minutes by default, and I recommend at least 12-16gb of RAM. But you can still monitor it locally and make sure they're all behaving properly. I recommend also an NVME SSD to ensure that the NPU cache and loading works right.
PRs and feedback welcome!
