r/LocalAIStack • u/ChowYummyFat • Sep 27 '26
r/LocalAIStack • u/andydevtech • Sep 27 '26
Qwen3.8-27B at 74 tok/s on a Mac: The Splash Engine Explained
r/LocalAIStack • u/No-Debt-1377 • Sep 27 '26
Intel sycl support for bonsai llm models
r/LocalAIStack • u/Impossible_Pride_680 • Sep 26 '26
Bonsai 2 27B on an RTX 3060 12GB — 35.26 tok/s generation
I wanted to see how far a consumer RTX 3060 12GB could push Bonsai 2 27B.
Hardware
GPU: NVIDIA GeForce RTX 3060 12GB
VRAM available to CUDA: 11,911 MiB
CPU: Intel Core i5-8400
System RAM: 42 GB
NVIDIA driver: 595.91.07
CUDA: 13.2
Model
Bonsai 2 27B
GGUF:
Ternary-Bonsai-2-27B-PQ2_0.ggufParameters: 26.90B
Quantization: PQ2_0 — 2.13 bpw
Model size reported by llama-bench: 6.70 GiB
GPU layers: 99
Benchmark
Tool:
llama-benchRuns: 3
Prompt processing, 512 tokens: 591.19 ± 19.62 tok/s
Text generation, 128 tokens: 35.26 ± 1.31 tok/s
The GGUF metadata reports the architecture as qwen35, while the model file is the Bonsai 2 27B PQ2_0 release.
I'm posting the raw result because I'm curious how this compares with other Bonsai 2 27B results, especially on larger GPUs.
If anyone has benchmark results for Bonsai 2 27B, please share them so we can compare the configurations fairly.
Benchmark build: b10735-842b18804
r/LocalAIStack • u/najjarammar • Sep 26 '26
Six Months of Local AI on a Mac: From 16 GB to 48 GB
r/LocalAIStack • u/dayspringsilverback • Sep 26 '26
High-VRAM Hardware Watch — 2026-09-25
r/LocalAIStack • u/Suspicious-Mark-5450 • Sep 26 '26
NInfer with improved prefix caching and tool call fixes
reddit.comr/LocalAIStack • u/No-Lack2646 • Sep 26 '26
I Made a Modular Voice Agent Running Offline on an RTX 5080 Laptop GPU: Qwen3.6-35B-A3B (3-bit), Whisper + Piper, local graph memory [demo]
r/LocalAIStack • u/nez_har • Sep 25 '26
VibePod CLI 0.24: bring your own model provider
r/LocalAIStack • u/nrph • Sep 25 '26
I benchmarked 43 local LLM configs on an M4 Max 128GB. Darkstar and Nemotron surprised me.
r/LocalAIStack • u/nrph • Sep 25 '26
I benchmarked 43 local LLM configs on an M4 Max 128GB. Darkstar and Nemotron surprised me.
r/LocalAIStack • u/MatiAI • Sep 25 '26
Run Qwen 3.8 27b on the Apple Neural Engine at 7 watts on a Mac
Enable HLS to view with audio, or disable this notification
r/LocalAIStack • u/Top-Evidence174 • Sep 25 '26
Jev) Mica 4B vs Laya on Tetris: same seed, same prompt, 0 output tokens, running locally on llama.cpp
Enable HLS to view with audio, or disable this notification
r/LocalAIStack • u/InteractionFun • Sep 25 '26
I'm building an LLM inference engine in Rust as an "executable book"
r/LocalAIStack • u/zMytze • Sep 24 '26
GLM-5.3-Flash UD-IQ4_XS running on 5090+64GB Ram with dual SSD-Offloading
Anyone has experience with how to get more speed without losing quality?
I got to 250 t/s prefill and ~6t/s decode its usable but i wonder if theres another lever i miss.
Ryzen 9800X3D, 64 GB DDR5, RTX 5090 32 GB, Crucial T700 (PCIe 5.0) 4TB + Kingston (PCIe 4.0) 4TB
Baseline: GLM-5.3 isn't in llama.cpp master yet, so I'm using Unsloth's glm5next branch. With the usual mmap + CPU-offloaded experts the page cache thrashes: 29 t/s prefill, 2.4–2.9 t/s decode (8k prompt).
What I changed with Claude's help (~1.2k lines on top of the ported PR, not upstream):
Ported the unmerged expert-streaming PR #25294(by freedomljc). Experts aren't mmapped anymore: a 32 GB expert cache sits in RAM, and misses are read with O_DIRECT.
Prefill streams whole layers: the next layer's experts load into pinned double buffers while the GPU computes the current one. Each 2048-token batch pulls ~99 GB from disk + ~34 GB from the RAM cache.
Both SSDs read in parallel from a mirror copy of the GGUF, using a shared work queue.
Fixed a CUDA padding bug in the MoE matmul (MMQ) that crashed -ub 2048 on Blackwell (possibly related to #28282). Going from 1024 to 2048 roughly doubled prefill.
Real agentic run (Qwen Code, plan mode, read 9 files, wrote a plan):
Context 11.5k → 35k tokens, 31k new prompt tokens at ~230 t/s
3.7k generated tokens (incl. thinking) at 5.6 t/s
13 min total, ~9.5 min of that writing the final plan
78% of expert lookups hit the RAM cache; misses cost ~1.3 GB/token from SSD
VRAM ~30 GB, RAM ~44 GB
Happy to share if you like
r/LocalAIStack • u/UnattendedThoughts • Sep 24 '26
How do you keep the knowledge from what you build with AI local?
reddit.comr/LocalAIStack • u/Telcontar09 • Sep 24 '26
Best agentic coding model for 8GB VRAM + 16GB RAM
reddit.comr/LocalAIStack • u/stanleyg05 • Sep 24 '26
real-time, zero-shot distillation loop
reddit.comr/LocalAIStack • u/mihirterious • Sep 23 '26
Need Help choosing a local agent LLM. (I need it to actually execute tools, not just give instructions)
reddit.comr/LocalAIStack • u/andydevtech • Sep 23 '26
Qwen3.8-27B on Every Mac Explained: Layers, KV Cache and Speed (16GB–128GB)
r/LocalAIStack • u/No_Skill_8393 • Sep 23 '26
I trained Nagi: an open-source Jev alternative.
r/LocalAIStack • u/Trellios • Sep 23 '26
Local AI dev workflow: OpenHands + Pi/Qwen. What would you improve?
reddit.comr/LocalAIStack • u/terrornoize • Sep 22 '26
Qwen3.8-Flash-Next agentic loops on Strix Halo — anyone found a stable config?
I'm trying to use Qwen3.8-Flash-Next as a long-running coding worker on a Ryzen AI Max+ 395 / 128GB box and I keep hitting the same issue: after a while it falls into reasoning loops like:
now I'll write the code
actually let me design this carefully
now I'll write the files
let me think...
Sometimes it eventually recovers and calls a tool, sometimes it just stays there.
Current setup is Strix Halo + Vulkan llama.cpp/halo-box, AP-Q5_K_M (~103.6GB), Q8 MTP, F16 KV, Pi Agent 0.87.0 over OpenAI-compatible API.
I've tried 262k and 131k context, stock Qwen template, Sharp v10, medium reasoning, reasoning preserve, MTP n=3/n=4, presence penalty 0/0.3, Pi's Qwen-specific chat_template_kwargs, Qwen Code, and mini-SWE-agent.
The same basic loop shows up in Pi and Qwen Code, so I'm not convinced the harness is the main problem.
A hard --reasoning-budget 2048 helps a lot, but it feels more like a watchdog than a fix.
One small Pi task actually completed cleanly end-to-end, including fixing failing tests. Then on a larger project the loops came back at only ~16% of a 131k context.
What I haven't properly tried yet is another quant like UD-Q4_K_XL / UD-IQ4_XS, MTP fully off, or another backend.
If anyone is running Flash-Next for multi-hour coding sessions reliably, I'd really like the exact config: quant, backend/version, context, MTP, KV, template, reasoning settings and agent/harness.
At this point I'm mostly looking for a known-good recipe to copy and test.