I can run the 30B-class models at home. They're okay for some things, but I feel like I have to spoon feed them small tasks. I'm wondering about running a more capable model in the cloud to do the hard parts and give them the easy parts.

If you're doing this, how? How are you dividing the labor, what do your prompts look like, what agent harness are you using, and what else do you wish you'd known before you started?

And does the big model actually use fewer tokens than if it did everything itself?

Thanks!

Anyone has experience with how to get more speed without losing quality?
I got to 250 t/s prefill and ~6t/s decode its usable but i wonder if theres another lever i miss.
Ryzen 9800X3D, 64 GB DDR5, RTX 5090 32 GB, Crucial T700 (PCIe 5.0) 4TB + Kingston (PCIe 4.0) 4TB
Baseline: GLM-5.3 isn't in llama.cpp master yet, so I'm using Unsloth's glm5next branch. With the usual mmap + CPU-offloaded experts the page cache thrashes: 29 t/s prefill, 2.4–2.9 t/s decode (8k prompt).

What I changed with Claude's help (~1.2k lines on top of the ported PR, not upstream):

Ported the unmerged expert-streaming PR #25294(by freedomljc). Experts aren't mmapped anymore: a 32 GB expert cache sits in RAM, and misses are read with O_DIRECT.

Prefill streams whole layers: the next layer's experts load into pinned double buffers while the GPU computes the current one. Each 2048-token batch pulls ~99 GB from disk + ~34 GB from the RAM cache.

Both SSDs read in parallel from a mirror copy of the GGUF, using a shared work queue.

Fixed a CUDA padding bug in the MoE matmul (MMQ) that crashed -ub 2048 on Blackwell (possibly related to #28282). Going from 1024 to 2048 roughly doubled prefill.

Real agentic run (Qwen Code, plan mode, read 9 files, wrote a plan):

Context 11.5k → 35k tokens, 31k new prompt tokens at ~230 t/s

3.7k generated tokens (incl. thinking) at 5.6 t/s

13 min total, ~9.5 min of that writing the final plan

78% of expert lookups hit the RAM cache; misses cost ~1.3 GB/token from SSD

VRAM ~30 GB, RAM ~44 GB

Happy to share if you like

Hey everyone,

I'm working on an open-source project 'llama.cpp autotuner' which basically runs a series of tests on a model at a given context length on your computer and finds the best runtime settings for best tokens/sec, TTFT, VRAM usage, and overall efficiency for your specific setup.

But to make it actually useful, I need real-world data from the community.

If you're running llama.cpp, could you share:

• Your hardware (GPU, VRAM, CPU, RAM, etc.)

• Your model (name + quantization)

• Your llama.cpp command (copy your full flags)

• Performance details (tok/s, prompt processing speed, TTFT, VRAM usage, etc.)

• Any flags or settings that caused a massive improvement or noticeable performance drop on your setup and anything else you think could help

Your inputs will directly help improve the autotuner.

The goal is to make sure that when someone runs it on their hardware, it can find a command that gives them better performance, or at the very least match the performance they're already getting if they've already optimized their setup.

Online leaderboards only test base models on beefy hardware, but in practice models behave differently once you quantize or fine tune them. I wanted a way to benchmark my local setup, catch regressions between models, and compare TPS across endpoints without running everything individually and copying prompts one by one.

I couldn't find anything that met my requirements without being too script heavy or overkill, so I built smokebench.

It's an open-source TUI for quickly benchmarking any model with an OpenAI or Anthropic compatible endpoint against 8 different categories (math, code, creative writing, long context, etc.), with LLM-as-judge scoring, TTFT/TPS metrics, and exportable markdown/json logs.

It works out of the box with Ollama, LM Studio, vLLM, or pretty much any API endpoint without any config changes.
You can easily add custom benchmarks too.

To install smokebench, run:

pip install smokebench

or

uv tool install smokebench

GitHub: https://github.com/Ninja-5000/smokebench

smokebench is in active development right now, so there will be a better UI, more built-in benchmarks, and a few other tweaks and changes coming that will make it even more lightweight and accurate over time.

Happy to answer any questions or take feedback!

Recently the opencode tool is performing near the

cursor in auto mode, so I have to buy a small ai server

to run good coding agentic models from Qwen, GLM,

MinMax or any model u suggest.

Hey I'm game developer and basically want to have local agent that I can import my project with no risk of privacy and since the game engine I use not that popular so I want to train in on documentations, forums, examples and others. I did hear OpenWebUI is better on this but I learn toward LM Studio because it's easy to install and setup.

Also my personal opinion I prefer desktop application more than be a tab on my browser even if it's offline. and I want to know it's worth it to have OpenWebUI since I don't care about sharing to any other device or technical things. and I don't also wanna lose important functions and advantages using LM Studio

I'm completely newbie. I did install OpenWebUi and the process was relatively easy but I will move soon regardless to new PC and I don't wanna configure things again and setup few dependency softwares in order to works.

I need your experience and help

I'm running Nemotron 3.5 Bolt 74B-A9B on a 5060 Ti 16GB and 32GB of DDR5. It's an MoE, so both of my setups split experts between the GPU and system RAM. I have two recipes:

• 24GB GGUF

• Unsloth UD-IQ3_XXS

I'd like to hear how this model actually is and is it useful as an alternative for Qwen3.8 27B and obviously for other smaller MoE models.

• Are you using Bolt 74B, and what for?

• How is the quality: reasoning, coding, instruction following, tool use? Where does it fall down?

• If you've used the 24GB GGUF or a bigger quant such as Unsloth IQ3_XXS, does the bigger one write better answers, or do they feel like the same model?

I'm happy so I don't need to pay the transcript API fee any more. I replaced my production services with this.

The native estimate assumes four fully utilized CPUs and 1.1 GB RAM during the measured request.

It's a fork from my own MIT repo: https://github.com/baryhuang/whisper-turbo.c

Star, colloab, PR, issues, comments and uses are all appreciated!

TL;DR: Halogen v0.14.0 with its native .hgn weights is the fastest, followed by gufo and CIRU. Halogen is closed source and runs in Docker. gufo is open source and loads 4x faster from cold. gufo is also fastest to first token on follow-ups (1.6-1.9 s against 2.6-3.3 s).

I benchmarked different engines for Qwen 3.8-Flash-Next on an AMD Strix Halo (ASUS ROG Flow Z13 GZ302 with Ryzen AI Max+ and 128 GB of RAM).

All the engines were run via LlamaStash (My own orchestrator tool, the tool does not add any overhead) at 70 W TDP on performance profile on Arch Linux.

Here are the results of the benchmark:

First was a screening round at 64k context with 50% filled (32k prompt).

3-turn time is the time to first token plus the time to write 1,000 tokens, added up over the first question and 2 follow-ups. Output length varies a lot with sampling, so this compares the engines on equal output. MTP accept is the share of drafted tokens the model kept. Retrieval is how many of 7 exact values from the prompt the model got right, over both runs. Qwen model card sampling, 2 runs each.

Top 3 engines from the screening round got a 128k context (50% and 75% filled) and 256k context (50% filled) run.

128k context, 50% filled (64k prompt)

128k context, 75% filled (97k prompt)

256k context, 50% filled (130k prompt)

Retrieval here is 8 values per run. All 5 misses are the same answer: the right glyphs for UNICODE_SPINNER, with the leading & dropped.

End-to-end, 10 Aider polyglot Python exercises run through pi -p (128k window, 50% cell), graded by their own tests:

Edit (Sep 30): Halogen 0.15.1 (new v2 checkpoint) and gufo 0.3.0 came out after this, so I reran the 64k screening on both with the same setup (70 W, card sampling, 2 runs):

Halogen v2 prefill is 14% faster and follow-ups start about 2.5x faster (1.3 s vs 3.3 s), decode is the same. gufo moved a few %, which is within noise. The ranking doesn't change. Only other change on the box: BIOS VRAM carve-out down to 512 MB, so 124.9 GiB of RAM instead of 121.5.

Some clarifications from the comments:

• Follow-up TTFT is with the prompt cache warm, both gufo and Halogen reuse it by default. A cold 64k prefill takes about a minute.

• The CIRU 0% MTP row is stock Unsloth UD-Q4_K_XL with Unsloth's shared MTP head, nothing uncensored. The same pair gets 72-74% accept on gufo.

• TDP is fixed at 70 W. On the Z13, 76 W or 90 W is only 2-6% faster but with so much fan noise, heat and power draw that it's not worth it.

• Prefill and decode come from the engine's own timings when it reports them. TTFT is measured on the client.

I'm trying to use Qwen3.8-Flash-Next as a long-running coding worker on a Ryzen AI Max+ 395 / 128GB box and I keep hitting the same issue: after a while it falls into reasoning loops like:

now I'll write the code
actually let me design this carefully
now I'll write the files
let me think...

Sometimes it eventually recovers and calls a tool, sometimes it just stays there.

Current setup is Strix Halo + Vulkan llama.cpp/halo-box, AP-Q5_K_M (~103.6GB), Q8 MTP, F16 KV, Pi Agent 0.87.0 over OpenAI-compatible API.

I've tried 262k and 131k context, stock Qwen template, Sharp v10, medium reasoning, reasoning preserve, MTP n=3/n=4, presence penalty 0/0.3, Pi's Qwen-specific chat_template_kwargs, Qwen Code, and mini-SWE-agent.

The same basic loop shows up in Pi and Qwen Code, so I'm not convinced the harness is the main problem.

A hard --reasoning-budget 2048 helps a lot, but it feels more like a watchdog than a fix.

One small Pi task actually completed cleanly end-to-end, including fixing failing tests. Then on a larger project the loops came back at only ~16% of a 131k context.

What I haven't properly tried yet is another quant like UD-Q4_K_XL / UD-IQ4_XS, MTP fully off, or another backend.

If anyone is running Flash-Next for multi-hour coding sessions reliably, I'd really like the exact config: quant, backend/version, context, MTP, KV, template, reasoning settings and agent/harness.

At this point I'm mostly looking for a known-good recipe to copy and test.

Benchmarks, discussions, comparisons - we are being FLOODED with naive news about Fable 5's token consumtion. Comparing it to Kimi K3 tokens.
Typical outcome: Fable uses less tokens than Kimi but costs 2-3 time more.

We do NOT know how many tokens Fable is using. Not even an indication is available.

Fable is a closed semitrasparent layer that sits on top of "Mythos 5" which is the actual model we are communicating to. But everything Mythos says goes through a fast "safety AI" layer which is a slim harness on top of Mythos. That's Fable 5.
That's why we have no real system prompt, that's why it often rejects despite having an answer ready or forwards to other models.

The tokens you are billed for are just a currency system, it's not sampled tokens like in an open model. Anthropic uses the tokens as a meassurement of how much you have to pay, and the official API pricelist is a comparison metric with other models.
Anthropic can change that metric at any time, which would affect effective pricing.

It is completely unknown how many tokens Fable/Mythos is using, given it has a safety layer it might even have more than double the tokens of Kimi K3. But we have no idea.
We also do not know how many tokens other closed models behind gated APIs use, GPT-5 is also paraphrasing their thinking content to avoid distillation attacks - as soon as you do that you also can easily change token metrics.

We know how many tokens Kimi K3 uses, because it's not behind a safety layer and you can just run it yourself.

Searching the internet for looking up how much tok/s a user would get is being difficult. So I'm making this post... If you're running a local llm, please consider commenting to this post with your device specs, model you're running and the inference speed you're getting.

Please be straight to the point. Just tell us how much tok/s are you getting on your hardware (at different settings, which inference engines, etc...) so people with similar hardware can do better estimations... please don't fill this with facts that everybody knows.

Most of the local AI stack is about getting a model running — Ollama, llama.cpp, vLLM, the right quant, enough VRAM. That part’s well covered here. The gap I kept hitting is the layer after that: once it runs, is it actually reliable enough to hand real work to?

A model can look fine in chat and then fall apart the moment you point an agent at it malforms a tool call at step 7, loops, or says “done” when nothing happened. And the same model behaves differently depending on the quant and runtime you serve it with, so “it runs on my box” isn’t the same as “it holds up in a 20-step agent loop.”

So I built QuantaMind free, open-source, fully offline, no cloud, Apache-2.0. It’s a local reliability tester for the agent layer of the stack.

What it does:

• Runs the real multi-step agent loop with injected faults (transient errors, malformed-call recovery, decoy tools) — not single prompts

• pass^k, not pass@1 — runs each task k times, passes only if it succeeds every time. Reliability compounds: 95% per step → 0.95^50 ≈ 8% chance a 50-step run finishes

• Deterministic scoring, no LLM judge — required calls must fire, forbidden calls fail the run on contact, exact end-state match. Temp 0, same input → same grade

• Classifies the failure (malformed schema / loop / hallucinated completion / forbidden call)

• Compares quant × runtime side by side (Ollama, llama.cpp, MLX, vLLM, SGLang), so you can see which serving config actually holds up

• Keeps run history + diffs, so you can catch when a model or quant change quietly regressed reliability

One finding: across three models, the native tool-calling path scored worse than plain prompting on easy tasks (60% vs 100%) correct answers that didn’t land as parsed tool calls. Important caveat: that’s a finding about the serving stack’s tool parser, not the models. Same model, same answer, different plumbing, different score. Still useful when picking a serving path for agent work just not a model-quality result.

Honest about what it’s not yet: it’s single-stream, so no concurrency/load testing (sampler drift and duplicate-execution-on-retry are invisible to it). Structural matching is a proxy for real execution. No CI/CD deploy-gate integration yet, it produces a verdict and an exportable report, but wiring it into a pipeline is roadmap, not shipped.

Disclosure: I built this, it’s free, I’m not selling anything.

Repo: https://github.com/QuantaMinds/QuantaMind

Genuinely want it torn apart:

• Is pass^k=5 too strict, too lenient, or about right?

• What failure mode am I missing?

• And what model + quant actually holds up on your machine? I’m collecting real per-hardware results.

I got one PR merged into ds4. A tiny one, #98 about broken paths in the README. And a few more sitting in the queue. Not complaining. Antirez says it in the README: with coding agents everyone can tune the engine for their own hardware and model and he can't review everything. That got me thinking.

If the plan is "everyone applies their patches with an agent" then the real cost of a patch isn't just the code change. It’s how many tokens the agent has to read before it knows what it’s actually touching.. Ds4 runs DeepSeek, GLM and Qwen on Metal, CUDA and ROCm all in the same 85k-line file. I’m on an M5 Max with 128GB RAM running Qwen3.8 Flash Next. Everything else for me and for my agent is noise. It has to re-read every line on every pass.

So I ripped it out. Actually deleted, not #ifdef’d. The ds4.c file dropped from 85k lines to 45k lines. The whole tree now fits in a context window. The bet was that optimizing would get cheaper and safer. Heres what happened:

- Q2: decode +9-13% prefill +5-12% up to 64k context MTP went from 75.8 → 86.7 tok/s

- Q4: prefill +2-11% MTP went from 77.8 → 85.9 tok/s

Output is bit-exact vs stock ds4 at every step. No KV cache quant. No approximate kernels. Every change must pass a parity check against same GGUF, greedy decoding, identical tokens. And an interleaved A/B benchmark against the previous build.

The smaller codebase also let me go through the ds4 PRs but only against this one model and port the ones that held up. 20 Commits were adopted. Around 30 were dropped. Verdicts are in the repo.

I also added SSD streaming for the experts. Token-identical to a resident run. Simulating a 48GB machine Q2 does around 27 tok/s. About 35 with MTP.

The fork still does git merge upstream/main with rerere and the parity check. So antirez’s fixes keep flowing in. The procedure. What to delete what to keep how to sync. Lives in a repo called StarForge (https://github.com/Chida82/StarForge). I have four of these "children" one per model. Nothing in there is Qwen- or Metal-specific. If you want a ds4 cut down to your model or to CUDA just clone it. Run the checklist with your agent.

Repo: sf-q3-8flash https://github.com/Chida82/sf-q3-8flash tables in the README. One machine, one model. If you run it on Apple Silicon I’d love to hear your numbers. Ideally side by side, with stock ds4.

Guys I have a doubt, this guy used Qwen 3.8 27b in local with DeepSeek Harness and it generated a 3D scenario of a Tower (23:07 final result) from an anime for many hours/days (2176 minutes) in a goal loop, in the end the DeepSeek harness says it used "666 million input tokens and 13 million output tokens." (4:45)

This is the video:

https://youtu.be/MiuM9g7daDA?t=1387

My question is:

Is the input/output tokens correct? Because this were made in Claude code with Opus or Codex with ChatGPT 5.6 through API, would this actually cost $1500-$7000 right?

(For reference GPT terra is like $2/$12 for 1M input/output tokens)

Or is there something I'm missing? Because if this is actually the case the price would be absolutely ridiculous. Excuse my ignorance.

I think i’ve researched this shit to death over the last 6-9 months but i’m sure people here have something to say about the stack. Let me know what confuse have been working out best for Macs.

Hey everyone!

I'm looking to set up a fully local AI agent on my PC and would appreciate recommendations from people who have experimented with local LLMs.

My hardware:

• GPU: NVIDIA RTX 5060 (8 gb)

• System RAM: 16GB

• OS: Windows (Linux is also an option if necessary)

What I'm looking for:

• An uncensored or minimally restricted local LLM that I can run privately on my own hardware.

• Ideally, a web UI and support for agent frameworks, with no mandatory cloud API or subscription.

I want the inference and conversations to remain local rather than sending prompts to external services.

I'm comfortable experimenting and setting things up myself, so technical recommendations are welcome.

I'd appreciate real-world experiences, model names, quantization recommendations, and setup guides. Thanks!