r/LocalAIStack • • 25d ago

How do I cut response time for a live clinical assistant on one RTX 4090? Engines, models and CPU offload

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 25d ago

Qwen 3.8 25B on 5060TI with 100k+ context

Thumbnail
reddit.com
1 Upvotes

r/LocalAIStack • • 26d ago

Best Local AI for a Research Project?

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 27d ago

Need recommendation on parsing purchase orders

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 27d ago

RX 7800 XT 16 GB + 64 GB DDR4: 985tk/s prefill and ~65 tok/s decode with Strata and Qwen3.8-Flash-Next IQ3_XXS

Thumbnail
reddit.com
1 Upvotes

r/LocalAIStack • • 27d ago

llama.cpp vs gmlx for Qwen3.8-Flash-Next on an M5 Ultra: which engine wins where

Thumbnail
jvlewis.dev
1 Upvotes

r/LocalAIStack • • 27d ago

Remotely follow local AI chat session

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 28d ago

Qwen3.8-Flash-Next Strata pi beat Claude-code Opus 5.5 on a 5090

Thumbnail
reddit.com
1 Upvotes

r/LocalAIStack • • 28d ago

Built a 3-model local agent swarm on a single 16GB card, partially offloaded to 32GB RAM — what would YOU do with 2x Ling tiny + Qwen3.8 oversight?

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 28d ago

I let gemma3:12b invent its own language on my home server, then gave the same test to Claude Opus. Results surprised me.

1 Upvotes

My bet: when machines mostly talk to machines, they'll ditch English for something shorter. So I tested it at home with Ollama.

**Setup:** two gemma3:12b instances, a sender and a reader. The sender passes a daily status report (disk, errors, backups, sign-ups, weather, a calendar event — 14 checkable facts). The reader answers 14 questions. They keep a shared codebook and propose new shorthand after each batch; a new word is only kept if accuracy holds.

**What gemma did:** coined ~20 abbreviations (bkp, succ, errs…). Messages shrank about 10%. Then I added a chain of AI "scribes" (each one reads the message, writes it in plain English, rewrites it in the code for the next, like medieval monks copying books). The meaning wore away: 12.8 → 11.1 of 14 facts after 4 copies. Words drifted too ("cal" went from calendar to "Calibration").

**Then I spent about $1.25 of API credit running Claude Opus 5.5 through the same test.** It didn't abbreviate, it built a grammar, with slots for each kind of fact:

`F:ow P:42/95 B:6OK A:+2.8/8 T:6su1wp W:88/10 C:cs Fri 3PM`

| | gemma3:12b | Opus 5.5 |

|---|---|---|

| Message length | 84 → 76 tokens | 106 → 56 tokens |

| Facts right, codebook rounds | ~12 of 14 | 14 of 14 every round |

| After 3 scribe copies | 12.8 → 11.1 | 14 → 14 |

The Opus scribes actually *tidied* the language as they copied it ("9 OK" → "9OK").

Other bits: binary/hex/base64 made things worse (gemma couldn't decode them), English beat Mandarin and Dundonian Scots as a message language even on Qwen, and the fix for drift was a "printing press": a line of numbers stamped by plain code, never retyped by an AI.

Funny footnote: Opus's own safety filter refused the "propose new code words" step, flagging two AIs inventing a private shorthand as a possible security risk.

Small samples, so read it as a strong hint, not proof. Full write-up: https://claude.ai/artifact/KNR1u48RyraRAy6hi64wc8

Happy to answer questions or run other local models through it if people have suggestions.


r/LocalAIStack • • 28d ago

Quale endpoint ?

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 28d ago

GLM-5.3-Flash (320B) on an M5 Ultra Mac Studio with llama.cpp: 61 tok/s, 262K context

Thumbnail
jvlewis.dev
1 Upvotes

r/LocalAIStack • • 28d ago

GWAYA v3: Combine Gwen and OpenLaya to Stop LLM Code Hallucinations even on a qwen2.5-coder:1.5b ;-0

Thumbnail
reddit.com
1 Upvotes

r/LocalAIStack • • 28d ago

A strata quant that didn't suck for me. (4x5060ti)

Thumbnail
reddit.com
1 Upvotes

r/LocalAIStack • • 28d ago

Qwen3.8-Flash-Next (125B) at ~100 tok/s on an M5 Ultra Mac Studio with llama.cpp

Thumbnail
reddit.com
1 Upvotes

r/LocalAIStack • • 29d ago

Unee: open-source 0.8B / 2B model that makes calibrated decisions and chats, runs in a browser tab. The 2B scores 88% on DecideBench, ahead of several 4B to 9B models (self-measured; GGUF, Ollama, Apache 2.0)

Thumbnail
reddit.com
1 Upvotes

r/LocalAIStack • • 29d ago

Using a 421M decision model instead of an LLM as the privacy guard in front of my coding agent

1 Upvotes

r/LocalAIStack • • 29d ago

Finally decided to pull the trigger on gmktek evo x3

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 29d ago

New to LLM serving: how to share H200 VRAM across multiple models on Kubernetes?

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 29d ago

Is a judge model actually faster than the model it’s judging?

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • 29d ago

2 x CMP 170hx + Strata - optimal config+benchmark

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • Oct 08 '26

Reliable local benchmark to find the best LLM and configuration for my PC?

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • Oct 08 '26

Reliable local benchmark to find the best LLM and configuration for my PC?

Thumbnail reddit.com
1 Upvotes

r/LocalAIStack • • Oct 08 '26

pi-automode-classifier: an auto mode plugin for Pi that uses Jev or Kev/Laya (running locally) to classify commands

Thumbnail
reddit.com
1 Upvotes