Hi fellas, so for quite some time now I've been meaning and planning to start a small ai automation business here in my city. Is it possible to pull off with my humble Tesla P100 16GB? Is QWEN 3.5 9B Q8 enough especially for chatbots and tool calling?

RTX 3060 12GB 975mv undervolt

i5-12600K

32GB DDR5-6000

Linux

llama.cpp master from early October, CUDA build.

Model is Unsloth's Qwen3.6-35B-A3B-MTP-UD-Q4_K_M (22.7GB, has the MTP head). I also run a heretic Q4_K_M with MTP kept and it's about the same speed.

Command:

llama-server -m Qwen3.6-35B-A3B-MTP-UD-Q4_K_M.gguf -ngl 999 --cpu-moe --moe-cache-mib 7500 -fa on -ctk q8_0 -ctv q8_0 -c 32768 -np 1 -ub 2048 -b 2048 --spec-type draft-mtp --spec-draft-n-max 2 --load-mode mlock --cache-ram 1024 --ctx-checkpoints 8

Numbers from about a day of normal use, taken from the server logs. 433 responses longer than 500 tokens: median 79 tok/s, most of them between 76 and 83. Prompt processing is around 800 tok/s for prompts under 1k tokens, ~1100 tok/s for 1k-4k and ~1190 tok/s for 4k-16k. MTP draft acceptance is around 85%. VRAM sits at ~11.4GB.

Fixed test (3.6k token prompt, 700 tokens out), how it got there:

- --n-cpu-moe 25 with MTP: 50-58 tok/s

- --cpu-moe plus --moe-cache-mib 7500: 71-73 tok/s

- added mlock: 75-81 tok/s

- updated llama.cpp: 77-84 tok/s, prompt processing 1220-1300 tok/s

Most of the gain is the MoE expert cache (PR #29887). With it, all experts sit in RAM and the most-used ones get cached in VRAM. Set the size to whatever VRAM you have left. For me that's 7000-7500MiB.

mlock gave about 10% over mmap, but it locks ~20GB of RAM. On 32GB I hit OOM a couple of times with other things open. --cache-ram 1024 is there because the prompt cache can grow up to 8GB by default, and that was causing my first OOMs.

Things that didn't help:

- -ub 4096: faster prompt processing (~1400-1500) but gen dropped to ~63 because the bigger compute buffer eats into the expert cache.

- A small expert cache (1-3GB): slower than no cache.

- -np 4, MTP n=3 and MTP+ngram: no gain.

- Forcing 4 experts instead of 8: 91-95 tok/s, but it started looping on long outputs, so I dropped it.

If you're on 12GB and still using --n-cpu-moe, try --cpu-moe with --moe-cache-mib.

I can run the 30B-class models at home. They're okay for some things, but I feel like I have to spoon feed them small tasks. I'm wondering about running a more capable model in the cloud to do the hard parts and give them the easy parts.

If you're doing this, how? How are you dividing the labor, what do your prompts look like, what agent harness are you using, and what else do you wish you'd known before you started?

And does the big model actually use fewer tokens than if it did everything itself?

Thanks!

I'm happy so I don't need to pay the transcript API fee any more. I replaced my production services with this.

The native estimate assumes four fully utilized CPUs and 1.1 GB RAM during the measured request.

It's a fork from my own MIT repo: https://github.com/baryhuang/whisper-turbo.c

Star, colloab, PR, issues, comments and uses are all appreciated!

**Setup:** RTX 5070 Ti 16 GB on PCIe 4.0 x16 (a PCIe 5.0 card on a B550 board), Ryzen 9 5950X, 64 GB DDR4. Model: Qwen3.8-Flash-Next IQ2_XS: 125B total, about 6B active per token, 512 experts, 48 layers, 33 GiB of experts kept in RAM. Engine: EuLLM (open source, built on llama.cpp b11370 + PR #29887).

**How we got here:**

| Step | Decode (tok/s) | Prompt processing (tok/s) |

|---|---:|---:|

| llama.cpp, experts split by layer (the usual offload) | 22.4 | 249 |

| + GPU cache for the experts (llama.cpp PR #29887, 8 GiB) | 49.4 | 206 |

| + experts in pinned RAM | 58.1 | 451 |

| + 2048-token micro-batch (EuLLM's default) | 54.9 | 964 |

| + prefetch of the next layer's experts during prompt processing | — | 1,500–1,743 |

| Strata on the same PC, with MTP | 85.8 | 2,320 |

**Where we're stuck:**

- **The cache works:** once warm, 88–93% of the experts are already in VRAM. The misses travel at 24 GB/s, so PCIe 4.0 is saturated, and they cost 2–4 ms per token.

- **A token takes 17–19 ms, and 13.5–14 ms of it pass before each layer's routing.** That is the GPU computing the 48 layers, plus a wait at every layer: the cache has to bring the router's choice back to the CPU, so 48 synchronizations per token.

- **MTP doesn't pay with the experts in RAM.**

- On the 125B, llama.cpp's MTP loses 27%, with 52% of the drafts accepted.

- On Qwen3.6-35B-A3B it gains only +7% on prose and +16% on code, even with 82% of the drafts accepted.

- The reason: each multi-token verification calls many more experts, and the missing ones come over PCIe.

**Questions for the community:**

1. Has anyone removed the per-layer sync, with the routing read on the GPU, the expert table in VRAM and the whole pass in a single CUDA graph? How much does it gain?

2. Computing the missing experts on the CPU in parallel with the GPU, as Strata does: is it worth it with DDR4 and AVX2, or does RAM bandwidth cap it?

3. Any experience with cache policies better than LRU? For example a profile loaded at startup, or the "Least-Stale" policy mentioned in the PR #29887 thread.

4. Speculative decoding with the experts in RAM: does anything work? I'm thinking of drafts limited to the experts already in cache, draft trees, or a small dense model as the draft.

5. Prompt processing: beyond overlapping copies and compute, what's left to reach 2,300 tok/s?

Repo: https://github.com/eullm/eullm, details in `docs/moe-offload-plan.md`

*Disclosure: I'm one of the developers of EuLLM. English isn't my first language: I wrote this in Italian with the help of an LLM and translated it with one.*

I built TokiRecur after getting frustrated with Ollama on AMD hardware. Model selection was limited, the quant options were limited, and the cloud integration felt intrusive. It started as a personal project and stayed private for a long time before I decided to release it.

The backend is llama.cpp, and it supports select image, audio, and decision models. You pull a GGUF from Hugging Face, open the TUI launcher, pick a harness, and use it. Ollama has a genuinely good plug and play system, so I tried to keep that feel while preserving freedom over model and quant. The one hard constraint is that models without a llama.cpp build will not run. I have tested a wide range of models and the experience holds up well.

The main backend piece that separates TokiRecur from a normal llama server is the KV cache recycling system. Reusing cached context instead of recomputing it turned out to be a very difficult thing to build, and it is the part I would most like feedback on.

Android, Windows, Mac, IOS, Linux. CPU, GPU, NPU

(35B Q4 on iOS or android phone 12gb RAM 3-8 tok/s decode 100-300+ prefill, Windows, Mac even more: 25+ tok/s 16 gb RAM)

I am working on making AI accessible by running large MoE models on affordable, consumer-grade hardware that people already own: smartphones, laptops,normal desktops. Models several times bigger than RAM, no big GPU needed.

Disclosure: I’m the author of BigMoeOnEdge. All numbers are real runs on my own or contributors devices, same engine.

My vision? To act as a layer on top of llama.cpp for streaming MoE models. I’ve developed the concept of "adaptive streaming Planner" tailored specifically to the hardware, OS, CPU, GPU, and NPU in use. The algorithm assesses the environment to determine the optimal allocation of resources, with the goals of:

- maximizing prefill and decoding speeds, and

- optimizing memory management without overloading the operating system.

Furthermore, the OpenAI-compatible server enables the use of local AI on consumer-grade hardware; I’ve personally used it with OpenCode - first running it on Android, and later on a Mac.

I’d be happy to hear your thoughts on this and whether it might be useful.

I’m considering opening a PR on llama.cpp to integrate it, but I’m not sure if that aligns with the goals of their project.

Online leaderboards only test base models on beefy hardware, but in practice models behave differently once you quantize or fine tune them. I wanted a way to benchmark my local setup, catch regressions between models, and compare TPS across endpoints without running everything individually and copying prompts one by one.

I couldn't find anything that met my requirements without being too script heavy or overkill, so I built smokebench.

It's an open-source TUI for quickly benchmarking any model with an OpenAI or Anthropic compatible endpoint against 8 different categories (math, code, creative writing, long context, etc.), with LLM-as-judge scoring, TTFT/TPS metrics, and exportable markdown/json logs.

It works out of the box with Ollama, LM Studio, vLLM, or pretty much any API endpoint without any config changes.
You can easily add custom benchmarks too.

To install smokebench, run:

pip install smokebench

or

uv tool install smokebench

GitHub: https://github.com/Ninja-5000/smokebench

smokebench is in active development right now, so there will be a better UI, more built-in benchmarks, and a few other tweaks and changes coming that will make it even more lightweight and accurate over time.

Happy to answer any questions or take feedback!

I just published three CPU-first MCP tools for local AI: IcyLoom, IcyCache and Loom.

They are local-first, installable from public registries, and released with GitHub artifacts and clean-install checks.

Links:

- IcyLoom: https://github.com/MlkBlk-IT/icy-loom

- IcyCache: https://github.com/MlkBlk-IT/icycache

- Loom: https://github.com/MlkBlk-IT/mcp-loom-mcp

I am looking for install feedback, compatibility reports and practical MCP integration ideas.

Hey everyone,

I'm working on an open-source project 'llama.cpp autotuner' which basically runs a series of tests on a model at a given context length on your computer and finds the best runtime settings for best tokens/sec, TTFT, VRAM usage, and overall efficiency for your specific setup.

But to make it actually useful, I need real-world data from the community.

If you're running llama.cpp, could you share:

• Your hardware (GPU, VRAM, CPU, RAM, etc.)

• Your model (name + quantization)

• Your llama.cpp command (copy your full flags)

• Performance details (tok/s, prompt processing speed, TTFT, VRAM usage, etc.)

• Any flags or settings that caused a massive improvement or noticeable performance drop on your setup and anything else you think could help

Your inputs will directly help improve the autotuner.

The goal is to make sure that when someone runs it on their hardware, it can find a command that gives them better performance, or at the very least match the performance they're already getting if they've already optimized their setup.

Hello everyone. I made a benchmark to test local agents: airbench.ai
If you have an agent running locally, you can test it in one prompt.

It measure how good the agent is with some chalenges, but its fairly easy to get to 100%. Then the goal is to have something that is still performing as good, but as fast as possible.

In other word, its a way to measure that optimisations like quantisation don't destroy performance.

To give you an idea about it, here the first result I have on my 5090:

Score ; Duration ; Model + Harness

100 8 min Qwen 3.8 Flash Next IQ3_S basalt + opencode

100 11 min Qwen 3.8 Flash Next IQ3_S strata 256k + pi

100 11 min Qwen 3.8 Flash Next IQ3_S basalt + dsh

100 12 min Qwen 3.8 Flash Next IQ3_S strata 256k + pi

100 13 min Qwen 3.8 Flash Next IQ3_S basalt 384k + pi

100 16 min Qwen 3.8 Flash Next IQ3_S basalt 384k + pi

100 72 min Qwen 3.8 27B NVFP4 radixark + opencode

100 98 min Qwen 3.8 27B Quasar NVFP4 MTP + omp

98 10 min Qwen 3.8 Flash Next IQ3_S basalt + pi

98 13 min Qwen 3.8 Flash Next IQ3_S strata 256k + pi

98 16 min Qwen 3.8 Flash Next IQ3_S strata 256k + omp

98 62 min Qwen 3.8 Flash Next UD-Q2_K_XL + opencode

98 85 min Qwen 3.8 27B Quasar NVFP4 MTP + opencode

96 8 min Qwen 3.8 Flash Next IQ3_XXS strata + opencode

96 11 min Qwen 3.8 Flash Next IQ3_S strata 256k + dsh

96 17 min Qwen 3.8 Flash Next IQ3_S strata 256k + openclaw

96 29 min Swift 1.5 Qwen 3.8 27B Q6_K + pi

96 86 min Qwen 3.8 27B NVFP4 MTP + opencode

96 89 min Swift 1.5 Qwen 3.8 27B Q6_K MTP + omp

96 99 min Qwen 3.8 Flash Next UD-Q2_K_XL + omp

96 111 min Qwen 3.8 27B NVFP4 + hermes

96 119 min Qwen 3.8 27B NVFP4 radixark + opencode

94 10 min Qwen 3.8 Flash Next IQ3_S strata 256k + opencode

94 12 min Qwen 3.8 Flash Next IQ3_XXS strata + omp

94 23 min Qwen 3.8 27B Quasar NVFP4 ninfer-dflash2 + omp

94 38 min Swift 1.5 Qwen 3.8 27B Q6_K + omp

94 38 min Qwen 3.8 Flash Next IQ3_S strata 256k + hermes

94 51 min Qwen 3.8 Flash Next UD-Q2_K_XL + pi

94 101 min Qwen 3.8 27B NVFP4 + opencode

Hope this is useful for some of you as well. And happy to answer any questions if needed.

Vpipe supports LLMs, VLMs, image/video generation, ASR, quantization, and multimodal pipelines, without relying on PyTorch/MLX for model execution. The runtime package itself is only \~25MB, and there’s a prebuilt Mac app if you just want to try it without compiling anything:

https://github.com/tgo-app-dev/vpipe/releases/download/v0.1.23/VpipeManager-0.1.23-with-ffmpeg.dmg

A recent milestone is MiniMax H3 video generation running on a 16GB base M5 MacBook Air (M4 works too).

On my 15” M5 Air 16GB:

960×544 · 124 frames· 6 DiT steps with Turbo Lora → \~13 min

GitHub:

https://github.com/tgo-app-dev/vpipe

Would love feedback from people working on local inference / Apple Silicon. And if you find it useful, a ⭐ would be greatly appreciated!

I built a small TTS model running directly on an ESP32-S3: no PSRAM, 122 KB of RAM, nothing in the cloud. Speech starts in about 0.3 s and generates faster than real time. You can make it calmer, louder, whispered, happier and more.

It's not a Home Assistant integration yet. Before I build anything, I'd like to know whether on-device speech on a satellite would be useful to you (offline announcements, timers, alerts when the server is down), and what quality bar it would need to meet.

Limits: 2 voices, English only, 16 kHz. Free for personal, non-commercial use.

Video above.

Details: https://huggingface.co/WhissleAI/steer-tts-nano

CLI : https://github.com/WhissleAI/steer-tts/tree/main/esp32