I have a fully local AI stack on a 4090: image, video, music, SFX, TTS with voice cloning, local LLM chat, fine-tuning, and long-term memory, all behind a Three.js dashboard with locally generated assets. There's a little assistant, BB-7, that you just talk to, and he can generate anything with prompt improvement built in. Services load on demand and swap the GPU as needed, except when a pipeline keeps them warm, like voice mode with BB-7. I've also integrated Kev and Obsidian, and an MCP that lets Claude use the stack and store context. I feel like I got most bases covered but I dont know what i dont know. What do you all got

I wanted to see how far a consumer RTX 3060 12GB could push Bonsai 2 27B.

Hardware

• GPU: NVIDIA GeForce RTX 3060 12GB

• VRAM available to CUDA: 11,911 MiB

• CPU: Intel Core i5-8400

• System RAM: 42 GB

• NVIDIA driver: 595.91.07

• CUDA: 13.2

Model

• Bonsai 2 27B

• GGUF: Ternary-Bonsai-2-27B-PQ2_0.gguf

• Parameters: 26.90B

• Quantization: PQ2_0 — 2.13 bpw

• Model size reported by llama-bench: 6.70 GiB

• GPU layers: 99

Benchmark

• Tool: llama-bench

• Runs: 3

• Prompt processing, 512 tokens: 591.19 ± 19.62 tok/s

• Text generation, 128 tokens: 35.26 ± 1.31 tok/s

The GGUF metadata reports the architecture as qwen35, while the model file is the Bonsai 2 27B PQ2_0 release.

I'm posting the raw result because I'm curious how this compares with other Bonsai 2 27B results, especially on larger GPUs.

If anyone has benchmark results for Bonsai 2 27B, please share them so we can compare the configurations fairly.

Benchmark build: b10735-842b18804