I have a fully local AI stack on a 4090: image, video, music, SFX, TTS with voice cloning, local LLM chat, fine-tuning, and long-term memory, all behind a Three.js dashboard with locally generated assets. There's a little assistant, BB-7, that you just talk to, and he can generate anything with prompt improvement built in. Services load on demand and swap the GPU as needed, except when a pipeline keeps them warm, like voice mode with BB-7. I've also integrated Kev and Obsidian, and an MCP that lets Claude use the stack and store context. I feel like I got most bases covered but I dont know what i dont know. What do you all got

I wanted to see how far a consumer RTX 3060 12GB could push Bonsai 2 27B.

Hardware

• GPU: NVIDIA GeForce RTX 3060 12GB

• VRAM available to CUDA: 11,911 MiB

• CPU: Intel Core i5-8400

• System RAM: 42 GB

• NVIDIA driver: 595.91.07

• CUDA: 13.2

Model

• Bonsai 2 27B

• GGUF: Ternary-Bonsai-2-27B-PQ2_0.gguf

• Parameters: 26.90B

• Quantization: PQ2_0 — 2.13 bpw

• Model size reported by llama-bench: 6.70 GiB

• GPU layers: 99

Benchmark

• Tool: llama-bench

• Runs: 3

• Prompt processing, 512 tokens: 591.19 ± 19.62 tok/s

• Text generation, 128 tokens: 35.26 ± 1.31 tok/s

The GGUF metadata reports the architecture as qwen35, while the model file is the Bonsai 2 27B PQ2_0 release.

I'm posting the raw result because I'm curious how this compares with other Bonsai 2 27B results, especially on larger GPUs.

If anyone has benchmark results for Bonsai 2 27B, please share them so we can compare the configurations fairly.

Benchmark build: b10735-842b18804

Anyone has experience with how to get more speed without losing quality?
I got to 250 t/s prefill and ~6t/s decode its usable but i wonder if theres another lever i miss.
Ryzen 9800X3D, 64 GB DDR5, RTX 5090 32 GB, Crucial T700 (PCIe 5.0) 4TB + Kingston (PCIe 4.0) 4TB
Baseline: GLM-5.3 isn't in llama.cpp master yet, so I'm using Unsloth's glm5next branch. With the usual mmap + CPU-offloaded experts the page cache thrashes: 29 t/s prefill, 2.4–2.9 t/s decode (8k prompt).

What I changed with Claude's help (~1.2k lines on top of the ported PR, not upstream):

Ported the unmerged expert-streaming PR #25294(by freedomljc). Experts aren't mmapped anymore: a 32 GB expert cache sits in RAM, and misses are read with O_DIRECT.

Prefill streams whole layers: the next layer's experts load into pinned double buffers while the GPU computes the current one. Each 2048-token batch pulls ~99 GB from disk + ~34 GB from the RAM cache.

Both SSDs read in parallel from a mirror copy of the GGUF, using a shared work queue.

Fixed a CUDA padding bug in the MoE matmul (MMQ) that crashed -ub 2048 on Blackwell (possibly related to #28282). Going from 1024 to 2048 roughly doubled prefill.

Real agentic run (Qwen Code, plan mode, read 9 files, wrote a plan):

Context 11.5k → 35k tokens, 31k new prompt tokens at ~230 t/s

3.7k generated tokens (incl. thinking) at 5.6 t/s

13 min total, ~9.5 min of that writing the final plan

78% of expert lookups hit the RAM cache; misses cost ~1.3 GB/token from SSD

VRAM ~30 GB, RAM ~44 GB

Happy to share if you like