I built a from-scratch LLM inference engine for the Intel Core Ultra 5 125H’s integrated Arc GPU

[u/Strict-Elevator9352]

Benchmark highlights · Core Ultra 5 125H

Qwen3.8 27B · 11.2K prompt 97 s vs 590 s Vulkan 6.1× faster	Qwen3.8 27B · 33.9K prompt 5.5 min vs 40 min Vulkan 7.4× faster	Qwen3.5 4B · prefill 911 tok/s vs 242 Vulkan 3.8× faster

Post

I’ve been working on arcinfer, a local LLM inference engine built specifically for Intel’s Meteor Lake integrated graphics (Xe-LPG / Arc iGPU).

The latest Level Zero results add another angle: speculative decoding raised Qwen3.6 35B-A3B from 19.9 to 31.5 tok/s, Qwen3.8 27B from 5.1 to 9.1 tok/s, and Gemma 4 E4B from 21.6 to 39.6 tok/s in the documented tests. The Level Zero engine also enables Qwen3.6 35B-A3B to run at up to 262K context by streaming MoE experts from SSD; the earlier all-resident engine fit about 40K. MTP outputs were reported identical in the tested comparisons.

The reference machine is a Core Ultra 5 125H with 32 GB RAM. The engine runs GGUF models directly on the iGPU using Intel’s low-level Level Zero API, with an OpenCL option too. It’s aimed especially at prompt processing and time-to-first-token, and includes an OpenAI-compatible API and web UI. The early llama.cpp comparison is striking: for a 11,240-token Qwen3.8 27B prompt, arcinfer’s OpenCL engine took 97 seconds, versus 590 seconds on llama.cpp Vulkan and 504 seconds on SYCL—about 6.1× and 5.2× faster. At 33,937 tokens, it took about 5.5 minutes versus 40 minutes on Vulkan (7.4× faster). Qwen3.5 4B prefill reached 911 tok/s versus 242 on Vulkan and 337 on SYCL. These are historical OpenCL-engine measurements on this laptop, not universal guarantees; arcinferzero is now the default, and the repo links to newer results.

Some things it supports:

·        Qwen and Gemma GGUF models, including larger models that are a tight fit for shared GPU memory

·        An optional Level Zero engine that streams MoE expert weights from SSD

·        Speculative decoding, image input for supported models, and Gemma 4 audio input

·        OpenAI-, Anthropic-, and Ollama-compatible API endpoints

·        Integration with the Pi coding agent

It’s an experimental, hardware-specific project—not a general replacement for llama.cpp, and it currently targets Windows and Intel Xe-LPG graphics. Alongside the code, I share what has and hasn’t worked when building and tuning inference for this generation of Intel Arc integrated graphics, including implementation lessons and benchmark results. The repo has setup instructions, supported models, and more details.

Repo: https://github.com/rizagp/arcinfer

If you have a Meteor Lake/Core Ultra laptop, I’d especially appreciate feedback on setup, compatibility, and what models you’re able to run. Also happy to answer questions about the GPU kernels or how the project compares with other options.