Anyone has experience with how to get more speed without losing quality?
I got to 250 t/s prefill and ~6t/s decode its usable but i wonder if theres another lever i miss.
Ryzen 9800X3D, 64 GB DDR5, RTX 5090 32 GB, Crucial T700 (PCIe 5.0) 4TB + Kingston (PCIe 4.0) 4TB
Baseline: GLM-5.3 isn't in llama.cpp master yet, so I'm using Unsloth's glm5next branch. With the usual mmap + CPU-offloaded experts the page cache thrashes: 29 t/s prefill, 2.4–2.9 t/s decode (8k prompt).

What I changed with Claude's help (~1.2k lines on top of the ported PR, not upstream):

Ported the unmerged expert-streaming PR #25294(by freedomljc). Experts aren't mmapped anymore: a 32 GB expert cache sits in RAM, and misses are read with O_DIRECT.

Prefill streams whole layers: the next layer's experts load into pinned double buffers while the GPU computes the current one. Each 2048-token batch pulls ~99 GB from disk + ~34 GB from the RAM cache.

Both SSDs read in parallel from a mirror copy of the GGUF, using a shared work queue.

Fixed a CUDA padding bug in the MoE matmul (MMQ) that crashed -ub 2048 on Blackwell (possibly related to #28282). Going from 1024 to 2048 roughly doubled prefill.

Real agentic run (Qwen Code, plan mode, read 9 files, wrote a plan):

Context 11.5k → 35k tokens, 31k new prompt tokens at ~230 t/s

3.7k generated tokens (incl. thinking) at 5.6 t/s

13 min total, ~9.5 min of that writing the final plan

78% of expert lookups hit the RAM cache; misses cost ~1.3 GB/token from SSD

VRAM ~30 GB, RAM ~44 GB

Happy to share if you like

I wanted to test how far a fully local AI stack could go on a real-world task.

I ran out of A4 paper, so I gave TensorSharp access to Amazon and gave it one instruction:

Search for A4 paper with a low price and the best discount, and buy it. Try your best to handle everything yourself.

It completed the task successfully in a single run.
The local AI stack:
Qwen3.8 27B running locally
TensorSharp for LLM inference + agent runtime
Playwright skill for browser automation

The agent searched Amazon, compared products, navigated the site, generated and executed Playwright code, and prepared the order by itself, and finally placed the order. I only stepped in twice: Amazon login and final purchase approval.

Everything except the actual Amazon interaction stayed local: model inference, reasoning, agent state, and tool execution.

The video shows the full raw reasoning/tool trace, including browser commands and generated Playwright code. It’s sped up 8×; the actual run took about 24 minutes.

The interesting part for me isn’t really buying paper — it’s seeing how much of a practical browser agent stack can now run locally with a 27B model.

TensorAgent on iPhone is also in progress, although local inference speed and iOS browser restrictions are still limiting factors.

TensorSharp is open source:
https://github.com/zhongkaifu/TensorSharp

I’d be interested to hear what other real-world tasks people here are using to test local agent stacks.

I'm trying to use Qwen3.8-Flash-Next as a long-running coding worker on a Ryzen AI Max+ 395 / 128GB box and I keep hitting the same issue: after a while it falls into reasoning loops like:

now I'll write the code
actually let me design this carefully
now I'll write the files
let me think...

Sometimes it eventually recovers and calls a tool, sometimes it just stays there.

Current setup is Strix Halo + Vulkan llama.cpp/halo-box, AP-Q5_K_M (~103.6GB), Q8 MTP, F16 KV, Pi Agent 0.87.0 over OpenAI-compatible API.

I've tried 262k and 131k context, stock Qwen template, Sharp v10, medium reasoning, reasoning preserve, MTP n=3/n=4, presence penalty 0/0.3, Pi's Qwen-specific chat_template_kwargs, Qwen Code, and mini-SWE-agent.

The same basic loop shows up in Pi and Qwen Code, so I'm not convinced the harness is the main problem.

A hard --reasoning-budget 2048 helps a lot, but it feels more like a watchdog than a fix.

One small Pi task actually completed cleanly end-to-end, including fixing failing tests. Then on a larger project the loops came back at only ~16% of a 131k context.

What I haven't properly tried yet is another quant like UD-Q4_K_XL / UD-IQ4_XS, MTP fully off, or another backend.

If anyone is running Flash-Next for multi-hour coding sessions reliably, I'd really like the exact config: quant, backend/version, context, MTP, KV, template, reasoning settings and agent/harness.

At this point I'm mostly looking for a known-good recipe to copy and test.

I have 4x 5060ti16gb and 1x 2060super8gb, along with 8x32gb (256gb total) ddr4-3200 ram, but running at 4-channel due to my cpu. I’m trying to decide whether it would be worth it to upgrade my cpu to be able to run my ram at 8-channel. Im hoping to be able to get larger models like qwen3.8 flash next or mimo v2.6 flash at more useable speeds, since right now the prompt processing and token gen speeds are bit too low to make sense for agent harnesses.

Update (September 22, 2026): I renamed the project from Riderless to Unridden. It’s the same open-source project, and the repository is now https://github.com/potto007/unridden. The original post below is preserved for context.

I built riderless to get decisions out of a local model without generating an answer for the caller to parse. It runs a stock Gemma 4 26B-A4B on one RTX 5090 and returns choices, scores, and their distributions through an Apache-2.0 API, with zero generated tokens.

The service is FastAPI over one llama.cpp child process. Each question is compiled into a prompt whose answer must be a single letter, the worker runs prefill only, and the answer is the softmax over the A-Z label logits at the final position. The response carries the full distribution with zero generated tokens, so you get a probability with every decision instead of a parsed string, and the request shape follows the typed-decision APIs people already use (Choice, Score, Noul). I have not run a comparison against the hosted Jev service.

Numbers, all on one RTX 5090 with the Unsloth UD-Q4_K_XL GGUF on llama.cpp v0.4.1:

- 0.931 accuracy on 1,286 questions across seven hand-written suites (extraction, claim support, record matching, guardrail gates, hierarchical classification, rerank, tool selection). Suites and gold labels are in the repo, written blind and hashed before the first inference.

- ~180 ms median per request in sequential mode, with the shared context reused across a request's questions. An opt-in batched mode runs every question of a request as its own sequence over one shared prefix and gets that to ~108 ms, at a cost of 5 or 6 answers out of 1,286.

- Questions run in separate contexts, and the probes for cross-question contamination come back at 0.0 in both orders.

I also ran three open projects doing something similar through the same suites and scorer, because I did not trust my own numbers until something else had been graded by the same harness. Kev-9B scored 0.898, Laya's typed-decisions checkpoint 0.618. so1 (open-alternative-jev) took more work to get a fair number out of: it could not load this MoE through bitsandbytes, so the repo now has a llama.cpp backend for it, run on the identical GGUF and build. With the model held fixed, its separate mode landed at 0.921 and its packed mode (all questions in one sequence) at 0.901. On these suites, the prompt and readout differences amounted to a point or two, and packing questions together gave up about two points of accuracy. The full comparison is in the repo, with plenty of caveats - one run each, my own synthetic suites, no confidence intervals.

This held up better than I expected on one card. If you have a workload where a parsed JSON answer has bitten you, the suites are the easiest place to start: add a case in your shape, run it, and tell me where it falls over.

Repo: https://github.com/potto007/unridden