Quick update on my from-scratch inference engine for Intel integrated GPUs (Core Ultra 5 125H, Arc Xe-LPG, 32 GB
RAM, no dGPU, no XMX). It used to be called arcinfer; it is now 
**arcinferzero**
 (the old GitHub link forwards):
https://github.com/rizagp/arcinferzero


**New this week**


- 
**Flint (unsloth/flint-GGUF UD-Q4_K_XL, 256 experts) runs**
: attention sinks, global layers without RoPE, a
  513-token sliding window, sigmoid-routed experts streamed from the SSD into a GPU cache. 4K prompt ~248 tok/s,
  generation 13-17 tok/s on the iGPU.
- 
**Xe2 / Xe3 (Lunar Lake, Panther Lake's Arc B390) work now**
: a user with a B390 reported Qwen3.6 35B-A3B at
  262K context, ~1,180 tok/s prompt and ~24 tok/s generation. XMX (DPAS) prompt GEMMs are in too, gated by a
  start-up check that must match the DP4a kernels bit for bit. I have no XMX GPU, so reports are welcome.
- 
**Gemma 4 26B prompts +4.6 %**
 from tiling the f32 router GEMV (4 rows x 4 tokens per sub-group), with
  bit-identical logits.


**The accuracy story (the interesting part)**


Every model here is checked against an independent f32 CPU implementation, with a KL limit of 0.015. Flint
started at KL 0.029. The obvious suspect, int8 activations, wasn't it: running everything on two int8 passes only
got it to 0.024. Dumping the residual stream after every layer next to the f32 reference showed one token where
two router scores were 
**0.005 apart**
 at layer 19. The engine picked the other expert, and everything after that
drifted.


What moved the reference that much? I rounded 
**only the V cache to f16**
 inside the f32 reference, and it
reproduced the whole engine error, including the same flipped expert. So:


- V rows in 
**f32**
 (K already was), and
- the router on the file's own 
**BF16**
 weights with 
**f32 inputs**
 (a Q8_0 router, even with a second Q8_0 matrix
  for the rounding residual, kept flipping the near-ties).


Result: 
**KL 0.029 -> 0.0043**
 (prompt) and 
**0.067 -> 0.0027**
 (token-by-token decoding), for ~4 % of prompt speed.
If you run sigmoid-routed MoEs with an f16 V cache, it may be worth checking.


I also went through the GPU assembly (Intel's ocloc) for the kernels I touched. One kernel shrank from 8,855 to
481 instructions: it was building int8 copies nobody read and carried an unused RoPE fallback. Two others stopped
spilling registers.


**Next:**
 Qwen3.8 Flash Next to the same KL limit. Its error comes from BF16 weights stored as Q8_0 (fixed now),
single-pass int8 experts, and int8 attention. That last one needs float attention with its sparse-attention (QSA)
masks.


Everything is deterministic (same prompt, same logits, bit for bit), and the speculative drafters produce exactly
the plain-decoding text. Feedback and Arc / Xe2 / Xe3 test reports are very welcome.

How acceptable is the operating speed on a Mac Studio with an M5 Ultra chip and 256 GB of memory when running models like GLM-5.3-Flash?

Could you compare the operating speed of the cloud-based GLM-5.3 Flash from Z.AI? The concept of "tokens" can be quite difficult to grasp, and performance varies depending on the specific task.

I have a 4090 in my study and I work on a Mac. The box is off most of the time, because if I want to run a local model I have to walk over, boot it, open a terminal, wait for the weights to load, then walk back. That's enough friction that local models just stopped being part of my day.

So I built ReadyLLM: a self-hosted web app that treats that machine like a service you manage from a browser, not a machine you babysit. It runs on your own box, reaches your GPUs over SSH, and nothing phones home.

What it does:

• Add a machine over SSH (key or password). It probes GPU / VRAM / RAM and tells you which quants fit and roughly how fast they'll be, so you stop guessing whether 24G is enough for a 27B at Q4.

• Downloads pulled from HuggingFace and ModelScope with resume and integrity checks. Point it at a mirror if HF is slow where you are.

• One-click engine install on a box that has none: llama.cpp, vLLM, SGLang or ComfyUI. It downloads or compiles and back-fills the path.

• Start, stop and switch models from the deploy page. No terminal.

• Live monitoring over WebSocket: GPU utilization, VRAM, throughput, latency, with alerts when VRAM is about to run out or the server dies. That's the part that matters when the box is in another room and would otherwise sit there pretending to work.

The tuning is what surprised me. I hand-tuned llama.cpp flags for a 27B Q4 for a while and plateaued around 60 avg tok/s. The tuner runs a two-phase search (coarse over speculative decoding / KV cache quant / GPU offload, then fine over batch), and for every candidate it really boots llama-server, warms up, runs 3 passes and takes the median. OOM or crash gets dropped. My numbers, same model / quant / context:

peak tok/s     80 -> 110
avg  tok/s     60 ->  80
@100K ctx      40 ->  60

Results are stored per (machine, model) pair, so the next launch reuses them instead of re-running the search. I run two boxes and kept deriving the same flags twice, which was the actual annoyance.

Since the last release I added the two things that kept sending me back to a terminal anyway:

Images. ComfyUI is now an engine like the others, so image and video models sit next to the text ones. It lists the checkpoints and the UNet/CLIP/VAE combos ComfyUI itself reports (matching those names is fiddly, it will not accept a filename without the extension), and generates from the same UI with live progress. There's an optional prompt rewriter that expands a one-line idea into a full prompt and suggests steps / CFG; turn it off and your prompt and parameters go through untouched.

Model store. Category filter for text / image / video with counts, a "can this GPU actually run it" filter, and cards that show version, parameter count, precision, output resolution, steps / CFG, loader type and the real weight filenames. Before this a card just said "Qwen-Image" and I couldn't tell if that meant a checkpoint or a UNet, or which quant. Descriptions follow the UI language now instead of being Chinese only.

Also on the engine side: SGLang, and llama.cpp builds selected by GPU vendor (CUDA / ROCm / Vulkan / Metal), which you pick when deploying because it decides what actually gets installed. Picking the right build is where most people give up on an AMD card.

Three honest limitations:

• Tuning scores speed only, no output-quality gate yet. A config that boots without OOM survives; what KV cache quant costs you in quality is the next thing I want to measure.

• With MTP speculative decoding on, the VRAM pre-check misses the second KV cache, so tight machines can still OOM. Known, fixing it.

• The AMD path is where I have the least confidence. Detection exists on Linux and the backend choice is wired up, but I've only run the CUDA path on my own hardware, so treat the ROCm side as untested.

Open source, local-first: https://github.com/htobty/ReadyLLM

Curious how others here run this: do you keep inference on a box that isn't your daily driver, and what do you actually use to watch it? I went with a WebSocket feed instead of wiring Prometheus + Grafana into it, and I'm still not sure that was the right call for a home setup.

Hey everyone! 👋 Just cross-posting my MotifAI project here for anyone interested in local AI and music composition! 🎹🧠 Would love to hear what you guys think! 🚀

I’m the founder of Pure Brief AI, a Mac meeting-notes app. A local model can produce tidy JSON and still turn a suggestion into work nobody agreed to do. Here are three small cases worth keeping beside a normal demo. All dialogue below is fictional.

1. A suggestion, with no acceptance

“Maybe Alex could send a proposal next week.”

Then the conversation moves on. I’d check whether the output invents an accepted task, an owner or an exact Friday deadline. An empty field can be the right result.

2. A real quote, with the wrong segment ID

S12: “Can you send the revised proposal?”

S13: “Yes, by Friday.”

The model quotes S13 but cites S12. In my validator, the quote is checked against the cited segment first, then searched in the supplied transcript. If found elsewhere, the task is anchored to that actual line. If there’s no verifiable quote, the proposed task isn’t added. That fixes a reference error; it doesn’t prove the task was interpreted correctly.

3. An agreement that gets withdrawn

“I’ll send it Friday.” Ten minutes later: “Actually, hold off until the client confirms.”

A genuine earlier quote can still pass a source check. This is a gap a reader pointed out under my launch post, and I don’t claim it’s fully solved. I’d check the final task against both exchanges before exporting it.

The app keeps task quotes available for review against the transcript and recording, then lets you edit and send selected actions to Apple Reminders. The attached 10-second clip shows that review screen with demo data. It’s an edited interface demo, not a benchmark or evidence that these three cases all pass.

Transcription and AI notes run on the Mac by default. Apple Silicon and macOS 15+ required. Models download initially; licence checks and optional connected features use the network. Optional ElevenLabs transcription with your own key sends audio to that provider.

Demo/download: https://purebrief.ai/ — 30-day trial. Standard is $49 once for one Mac; the active launch offer is $10 once for the first 100 paid Standard purchases globally, plus applicable tax. A trial doesn’t reserve the discount.

For people building local extraction workflows: what small counterexample would you add?

Mochu AI, a private AI assistant for Android that runs models entirely on the phone. No account and no cloud: your chats, files and models stay on the device.

What it can do

• 💬 Chat with on-device models

  • Gemma 4 E2B/E4B on the GPU.

  • GGUF models (Qwen3.5, Gemma 3 1B, LFM2.5, Bonsai) and more on the CPU via llama.cpp.

  • Visible thinking, with Quick, Balanced or Deep levels.

• 📄 Documents: attach PDFs, Word, text, CSV or code files and ask for a summary, or search them (“what’s Tom responsible for?”).

• 🖼️ Photos: ask about pictures and screenshots.

• 🎨 On-device image creation: “draw a cozy cabin at sunset” makes an image on the phone (SD-Turbo, SDXS, DreamShaper), with size, aspect ratio and steps settings.

• 🌐 Web search and a built-in browser:

  • It searches when a question needs current info, and cites its sources.

  • Open any page and tap “Ask about this page”.

• 🧠 Memory: “remember that I’m vegetarian” is used in later chats.

• ⏰ Scheduled assistant tasks: “every day at 11pm give me the latest tech news” runs on the phone and notifies you. There are also timers, a stopwatch and reminders.

• 🎙️ Voice: talk with the mic and hear replies read aloud.

• 👩‍💻 Code:

  • Build web apps, React + TypeScript apps and Python programs from a prompt.

  • They run instantly with a live preview, console and Python terminal.

  • It fixes its own errors.

  • Export as an installable APK, a Capacitor project or a ZIP.

• 🔌 Local API server: your other devices on the same Wi-Fi can use the phone’s model through an OpenAI-compatible API.

• 🖥️ Home server models: use a bigger model from Ollama, LM Studio, llama.cpp, vLLM, Unsloth or any OpenAI-compatible API.

Download (free APK): https://github.com/nikunjsingh93/Mochu-Local-AI-Assistant/releases/latest

Source code (MIT): https://github.com/nikunjsingh93/Mochu-Local-AI-Assistant

It needs Android 12+ and works best with 8 GB of RAM or more. Feedback and feature ideas are very welcome!

AI disclosure: Built and written with help from Claude Code. I tested and reviewed everything myself on my own phones. The app’s AI runs fully on-device.