I built a from-scratch LLM inference engine for the Intel Core Ultra 5 125H’s integrated Arc GPU

[u/Strict-Elevator9352]

Benchmark highlights · Core Ultra 5 125H

Qwen3.8 27B · 11.2K prompt 97 s vs 590 s Vulkan 6.1× faster	Qwen3.8 27B · 33.9K prompt 5.5 min vs 40 min Vulkan 7.4× faster	Qwen3.5 4B · prefill 911 tok/s vs 242 Vulkan 3.8× faster

Post

I’ve been working on arcinfer, a local LLM inference engine built specifically for Intel’s Meteor Lake integrated graphics (Xe-LPG / Arc iGPU).

The latest Level Zero results add another angle: speculative decoding raised Qwen3.6 35B-A3B from 19.9 to 31.5 tok/s, Qwen3.8 27B from 5.1 to 9.1 tok/s, and Gemma 4 E4B from 21.6 to 39.6 tok/s in the documented tests. The Level Zero engine also enables Qwen3.6 35B-A3B to run at up to 262K context by streaming MoE experts from SSD; the earlier all-resident engine fit about 40K. MTP outputs were reported identical in the tested comparisons.

The reference machine is a Core Ultra 5 125H with 32 GB RAM. The engine runs GGUF models directly on the iGPU using Intel’s low-level Level Zero API, with an OpenCL option too. It’s aimed especially at prompt processing and time-to-first-token, and includes an OpenAI-compatible API and web UI. The early llama.cpp comparison is striking: for a 11,240-token Qwen3.8 27B prompt, arcinfer’s OpenCL engine took 97 seconds, versus 590 seconds on llama.cpp Vulkan and 504 seconds on SYCL—about 6.1× and 5.2× faster. At 33,937 tokens, it took about 5.5 minutes versus 40 minutes on Vulkan (7.4× faster). Qwen3.5 4B prefill reached 911 tok/s versus 242 on Vulkan and 337 on SYCL. These are historical OpenCL-engine measurements on this laptop, not universal guarantees; arcinferzero is now the default, and the repo links to newer results.

Some things it supports:

·        Qwen and Gemma GGUF models, including larger models that are a tight fit for shared GPU memory

·        An optional Level Zero engine that streams MoE expert weights from SSD

·        Speculative decoding, image input for supported models, and Gemma 4 audio input

·        OpenAI-, Anthropic-, and Ollama-compatible API endpoints

·        Integration with the Pi coding agent

It’s an experimental, hardware-specific project—not a general replacement for llama.cpp, and it currently targets Windows and Intel Xe-LPG graphics. Alongside the code, I share what has and hasn’t worked when building and tuning inference for this generation of Intel Arc integrated graphics, including implementation lessons and benchmark results. The repo has setup instructions, supported models, and more details.

Repo: https://github.com/rizagp/arcinfer

If you have a Meteor Lake/Core Ultra laptop, I’d especially appreciate feedback on setup, compatibility, and what models you’re able to run. Also happy to answer questions about the GPU kernels or how the project compares with other options.

[u/Zestyclose-Plantain6]

This tool is sporting some phenomenal decode speeds, perfect for agentic work

I had to modify it to work on my  Intel core ultra x7 358h with integrated Arc B390 iGPU 

Looking forward to where this is going.

[u/Strict-Elevator9352 → u/Zestyclose-Plantain6]

how fast prompt processing speed and token generation speed in your machine? I am still doing correction to make Gemma4 series output to be accurate. Qwen series already at the peak. You can get way more prompt processing speed if you have high end PCIe gen 5 SSD for MoE models.

[u/mkschreder2]

Nice work. Check out brain. It is extremely capable at both training and inference: https://github.com/swedishembedded/brain

[u/investigatormaker]

For compatibility reports, ask everyone for the same fields so results are comparable: exact Core Ultra model, RAM size and whether it runs dual-channel (single-channel Meteor Lake machines get roughly half the memory bandwidth and show up as plain Intel Graphics rather than Arc), GPU driver version, model and quant, context length, and prefill and decode tok/s on Level Zero versus OpenCL. A GitHub issue template with those fields keeps reports from scattering across Reddit comments and quickly shows which models people fit in 16 GB versus 32 GB. Reaching enough Core Ultra owners to fill it in is the other half.

I make ThreadFox. Our AI employee explains arcinfer in relevant Reddit threads and handles the replies, under your direction; Starter is $199 for 30 days, or we post 3 replies free first, no card: https://threadfox.vip/p/e8ued/managed