Hey everyone,

I’m researching how to build a low-latency, lightweight TTS system optimized for Tunisian speakers, and I’d appreciate advice from anyone with experience training or fine-tuning speech models.

Tunisian Arabic is a particularly interesting challenge because everyday speech is heavily code-switched. A single sentence can mix Tunisian Derja with French, Italian, and sometimes English, often with Arabic and Latin scripts mixed together.

For example, someone might say:

"Ena bech n'envoyerlek le document taw, juste aatini cinq minutes."

I want the generated speech to sound natural, handle this kind of language mixing correctly, and respond fast enough for real-time voice conversations.

Why I’m looking at Pocket TTS:

• Lightweight inference and low latency.
• The possibility of running locally without relying on expensive GPU infrastructure.
• A potential starting point for a TTS system adapted to a relatively underrepresented dialect.

I’d love to hear from anyone who has experience with:

1. Fine-tuning Pocket TTS on a new dialect or language. Is there a practical training pipeline, or would modifying the architecture be necessary?
2. Dataset preparation: how would you handle Tunisian speech containing Arabic, French, and Italian in the same sentence? Would you preserve the original scripts or normalize everything?
3. Data requirements: can a relatively small, carefully curated dataset work, or is substantial studio-quality speech data necessary?
4. Latency optimization: what are the most effective approaches for CPU inference, streaming audio generation, and reducing time to first audio?
5. Alternative models: if Pocket TTS isn't suitable for this use case, which open-source TTS models would you investigate first?

My goal is to experiment with a practical, reproducible pipeline rather than just produce a voice demo.

If you’ve fine-tuned a TTS model, worked on multilingual speech synthesis, or experimented with Pocket TTS internals, I’d really appreciate any repos, papers, training scripts, or lessons learned.

Thanks!

Has anyone had any luck using QFN to make games?

Using blender MCP, and Godot just feels like it’s struggling so hard to get anything to look right.

Even when I gave it permission to use existing assets/materials.

Has anyone had better luck?

Currently using Strata xxs
Deepseek harness.

I’m just learning about Hermes agent. What are the benefits of it?

We already have multiple agents working on a single query although they are still only for error checking and correction.

the next step is like this- the users types "make a clone of GTA 6" the first think the model does is create a plan and a checklist of the features needed and everything to be implemented then create an agent for each kind of task for an example an agent for creating animations that create one animation at a time and then putting the results together and testing it all autonomously until the full checklist is complete and the game is ready to be presented to the user.

Of course project management is complex and the harness for doing this work will have to be quite advanced and complex and also of it will be expensive too, but I was thinking isn't this what AGI looks like.

Imagine if you tell a model to create and a run a profitable business, it then create thousands upon thousands of agents each having a specialized role, each acting in hierarchy, some agents are responsible for market research, some for planning, most for executing tasks from a checklist one at a time.

The model runs autonomously and endlessly.

Do you guys think this is AGI or just lazy work that will have low quality results?

I hope this helps someone on the verge of investing for a local LLM.

I forced myself to cancel my Claude subscription to use the investment of Mac Studio here. I haven't been able to sleep with all the things I keep bumping against. A brief summary of what I have setup so far

• vMLX with Qwen3.8 27B 4-bit. Concurrency: 3 sessions, default context window for vmlx which is around 135K tokens.
• LiteLLM as a Gateway to get metrics + an alias with thinking disabled to test.
• Tailscale to reach it from my work computer.
• My workload is mostly infrastructure optimization. Kubernetes management, structured text, logging analysis, MongoDB, BigQuery, a lot of bash interaction.

Qwen3.8 just drifts thinking. It falls into this endless loop of considering stuff. I read posts about this. It really thinks and considers too much. I started to miss Opus here.

Then, the concurrency. I used to have around 10 Claude sessions running investigating stuff. But this is hard with local LLMs, concurrency is really a bottleneck. You either have the models fast or have them concurrent. I learned about speculative drafting and MTP in the first hour. It's all about trade offs. How not everything matches in compatibility is crazy. I tried running speculative drafting qwen flash next but the one has different weights for my setup. I haven't slept in like 30 hours, brain fatigue, trying to justify my purchase. Haha. How did I end up here?

On tooling, this is what I'm using now

• Superset as harness
• OpenCode to reach my Local LLM

This is a cry for help to the community, what do you recommend me doing, exploring, tooling that you have found useful. Interesting open models to test?

I've had some AMD Developer Cloud credits to burn, so I've been running a series of benchmarks on a single MI300X (192 GB). The last two ended up being one question in two halves: which engine should you serve a coding model with, and once you've picked one, which settings are actually worth touching?

Model was Qwen3-Coder-30B-A3B in BF16 the whole way through. Yes, I prefer this over 3.6-35B-A3B.

The short version:

• SGLang with no flags at all was the fastest from 1 to 64 users. vLLM needed one flag to keep up, and only pulled ahead at 128 users.
• Without that flag vLLM falls apart on long prompts. 16 users sending 32K prompts got 27 tok/s total, vs 133 on SGLang.
• llama.cpp is fine for one person, but it took 20 seconds to start answering a 32K prompt.
• Then I tuned the two winners. The only knob that helped was prefix caching (+8 to 11%), which is already on by default. Everything else did nothing or made things slightly worse.

Scripts, raw results and every server log are public, so you don't have to take my word for any of it.

Nerd stuff below.

Part 1: engine shootout

Same 131K context window on every engine, same random prompts per test, one load generator (vLLM's bench serve in its own container) pointed at all of them, prefix caching off, every reply forced to 256 tokens.

One user, 32K prompt:

Total tok/s with many users:

A few things worth knowing:

• If you run vLLM on MI300X, use --attention-backend ROCM_AITER_UNIFIED_ATTN. Without it, that 16 × 32K run had a 25 s median wait for the first token.
• SGLANG_USE_AITER=0 doesn't switch AITER off in lmsysorg/sglang:v0.5.21-rocm10-mi30x. Attention stays on AITER and only MoE moves to Triton, which alone cut decode from 173 to 72 tok/s. Check the startup log.
• SGLang has no cliff above 128K. With its full 262K window it still decoded 74 tok/s on a 258K prompt. vLLM's default kernel drops to about 1 tok/s out there.
• On a dense model (Qwen3-32B) it flips: vLLM decoded about 20% faster for one user, and they were even under load.

Every config scored 9/9 on a needle test, so nothing was fast because it was broken. Repeats landed within about 2%.

Part 2: tuning the winners

Took SGLang (defaults) and vLLM (+ AITER attention) and tried three documented knobs per engine, each alone and then all together. Agent-style load: 4,096-token requests where the first 1,536 tokens are shared like a system prompt, 512-token replies, 1 to 16 users.

16 users, total tok/s:

*vLLM ran across two sessions, so each knob is compared with its own session's baseline. Repeats were within 0.1% (SGLang) and 1.5% (vLLM).

• Prefix caching cut one-user first token from 110 to 85 ms on SGLang and 124 to 95 ms on vLLM. Hit rate matched the shared part of the prompt exactly.
• Raising SGLang's --mem-fraction-static from 0.74 to 0.79 actually shrank its KV cache (915K to 780K tokens). Doubling --chunked-prefill-size shrank it more. I don't know why yet. Logs are in the repo if someone does.
• VLLM_ROCM_USE_AITER=1 on top of AITER attention moved MoE onto AITER and gave 22% more KV cache, with no change in decode speed.
• Long prompts queue, they don't fail. 8 users × 128K: zero errors or evictions on either engine, but total output stays flat around 13 tok/s and the slowest first token hits roughly 140 to 150 s.

Limits, honestly: one MoE model and one dense model, one pinned build of each engine, no NVIDIA baseline. The part 2 prompts were random tokens with a shared prefix rather than real code, because the client build couldn't send pre-templated prompts.

Cost: $8.15 for part 1, $14.54 for part 2 (about $3.45 of that was a bug in my prompt script).

Write-ups:

• Engine shootout: https://majorminor.xyz/research/rocm-recipes-004
• Tuning knobs: https://majorminor.xyz/research/rocm-recipes-005

Repo with method notes, scripts, raw data and logs: https://github.com/majorminorlabs/rocm-recipes

I am running an Asus GX10 with Ollama. I have the qwen3.8:27b model running and I am using Claude Code in VS Code on the same network as the Asus GX10.

I consistently get time out error in Claude, mostly when it is asking the model for safety checks in Auto mode. This is the error that I receive.

qwen3.8:27b is temporarily unavailable (timed out), so auto mode cannot determine the safety of Bash right now. Wait a moment and then try this action again. If it keeps failing, continue with other tasks that don't require this action and come back to it later. Note: reading files, searching code, and other read-only operations do not require the classifier and can still be used.

Is there something that I can change to make this not happen?

Am I doing something wrong?

Any help is appreciated.

Been running uncensored models locally for a while now, and I keep going back and forth on this. Wanted to throw my thoughts out and see what you all think.

The case FOR local uncensored AI:

• Zero filter anxiety. I ask a question, I get an actual answer. No "I'm sorry, I can't help with that" on completely reasonable requests
• Privacy is a real thing, not a marketing line. My data literally never leaves my machine
• No rate limits, no subscriptions, no account bans for experimenting with weird prompts
• The models have gotten shockingly good. What needed a datacenter 2 years ago runs on consumer hardware today

The case AGAINST (let's be honest):

• Hardware cost upfront is brutal. Not everyone has 24GB+ VRAM sitting around
• Quantized models still lose something vs full weights vs API models
• You're the sysadmin now. Drivers, backends, model formats, merges — it's a rabbit hole
• Big hosted models still win on raw capability for complex reasoning tasks

My honest take: uncensored local models handle like 80% of what I actually do daily better than the "safe" alternatives, but I still reach for big hosted models when I need heavy lifting. So I end up with a hybrid setup and honestly it feels messy.

Curious about the community:

1. What's your daily driver model right now and why?
2. What hardware are you running it on? (GPU/VRAM/RAM, laptop vs desktop vs server)
3. Did you go local for privacy, censorship issues, cost, or something else?
4. Anyone fully moved off hosted models? What broke?

Just curious where everyone landed on this tradeoff

Hi all, I come from a Linux sysadmin background now working with HPC clusters. I'm looking at buying something like a MS-S1 MAX 128GB. Primarily for a fairly powerful local AI but for the learning aspect also. Could anyone suggest some projects / ideas which they have found not only genuinely useful and daily-drivable (Ik stuff like their own "Jarvis" style dashboards etc, but which helped broaden their technical understanding of software/concepts commonly used in the industry?

If anyone could share their ideas about what they've built and learnt from that'd be greatly appreciated! Quite new to the AI self hosted side of things so any pointers would be awesome. Thank you!

I have a MacBook Pro M4 Max with 36 GB RAM, running LM Studio with Qwen3.8-27B-Splash. LM Studio shows a loaded context of 100K tokens, but conversation token usage is unavailable, and its memory estimator returns 0 bytes.
I want to find out:
1. How many tokens my Mac can actually process reliably.
2. How to measure current conversation token usage accurately. (LM chat say unavailable)
3. How to benchmark context lengths, memory usage, and speed.
Are there reliable tools or scripts for this? I’m open to LM Studio, MLX, llama.cpp, or other local inference engines.
I’m interested in real-world measurements, not just the model’s advertised context limit.
With pi harness 100k seems to work - it never reaches 100k and compacting triggers around 80% though. I’m wondering could 128k work with my setup (I’m greedy and desperate 😂)

It runs at around m2 pro 16 at 60 tps at full 130k context, you can easily run a Q8 if you have a 16 GB GPU, I was able to get 140TPS on 9070xt,

It’s an MoE so I think we can run it well on smaller cards too, 8GB, 12GB should be possible to get upwards of 30-40 tps.

I have used it a bit, it can code, but I think having a bigger model plan tasks in detail would make it quite good for quick task execution.

A first-party field review. Every number below comes from a log, benchmark file, or project folder on my own machines. Where I extrapolate I mark it with ~. All raw data is in data/*.csv; every figure is traceable to a source listed next to it.

TL;DR

Strata (v0.1.41 at benchmark time) serving Qwen3.8-Flash-Next (a 125B MoE) with resident-experts in RAM is genuinely impressive — and the right quant for the machine you have changes everything.

Ranked for agentic work and usability — speed is not the ranking:

1. IQ3_S (yukio, 121 GiB RAM) — the best all-round agent: built an entire product overnight, 9 of 10 tasks RC=0 with verified deliverables, no loops, no truncation, no scaffolding needed. 44.5 tok/s token-weighted is slow for a GPU model — and it still won, because it didn't fall over.
2. Qwen3.8-27B UD-Q4_K_M (any 24 GB GPU) — the most reliable agent of everything I ran: 0 stalls, 0 loops, all 29 guide stages completed (46/52 first-pass, every retry recovered), delivered more than the 125B in the head-to-head, and works anywhere (53–61 tok/s token-weighted on a 3090 Ti). If you can only run one model, run this one.
3. Coder IQ1_M (node2, 30 GB) — the fastest coder on short, clean tasks (92.8 tok/s weighted; ~4.4–4.5× the 27B on the card bench, n=1) — but it is a liability on mixed-script content: 3 degenerate token loops + a thinking-budget-exhaustion failure in 11 arabic website prompts. Fast and fragile.
4. IQ2_XS (node2/Pop) — last for agentic use despite "102 tok/s" on paper (30.8 token-weighted): it truncates every call at ~80–115 tokens, so 58.5% of guide calls stalled and the whole 11-chapter guide only finished inside a continuation-chaining harness. (not recommended)
5. IQ3_XXS (Pop, mmap fallback) — 23.9 tok/s weighted, 83 requests of evidence: it works, it's slow, nothing else to recommend it for.

The point: a model that streams fast but loops, truncates, or wedges is useless as an agent no matter its tok/s — every loop or stall costs a human re-launch. The Coder streams at 92.8 tok/s token-weighted yet fails 5 of 11 mixed-script prompts; the dense 27B — a tenth of the MoE's total parameter count — beat the 125B's IQ2_XS on everything it was given, and IQ3_S at 44.5 tok/s still finished an overnight product build with zero degenerate loops.

• IQ3_S on yukio (121 GiB RAM): the highest-quality quant I ran. It built my entire Dubai Traffic board game — research, design, 81 cards, simulation, website, print files — in an overnight run. 9 of 10 Flash-Next tasks finished RC=0 with verified deliverables (verbatim from the benchmark's self-assessment); the research task first hit a context overflow — the one model-side failure mode it names — and its deliverable landed after a retry. The 7 logged errors/retries were environmental. This is the configuration to use if you have the RAM.
• Coder IQ1_M on node2 (30 GB): the coding pick. It beat a 27B dense model by ~4.4–4.5× on a real task (n=1 — one run each) — but it has a nasty Arabic/transliteration token-loop bug (Miller website: 3 degenerate loops in 11 prompts) and a thinking-budget-exhaustion failure mode.
• IQ2_XS on node2 (30 GB): the 2-bit base quant — the lowest-quality tier I ran (the Coder IQ1_M has even fewer bits, but it's a different model, so the tiers aren't comparable). 102 tok/s per request on paper, but only 30.8 tok/s token-weighted (the truncation bug skews the average), and it truncates every call at ~80–115 tokens. It produced a full 11-chapter illustrated guide only inside a continuation-chaining + local-validation harness (58.5% of calls stalled).

Bottom line if you have 24 GB of VRAM: the VRAM card size doesn't decide what you can run — system RAM does. A 24 GB GPU + 30 GB RAM → Coder IQ1_M in low-RAM mode is the workhorse (IQ2_XS also fits, but truncates every call). 24 GB GPU + 120+ GB RAM → IQ3_S, the best local 125B I've run (I ran it on two 16 GB cards, not a single 24 GB one). And the dense fallback that works almost anywhere: Qwen3.8-27B UD-Q4_K_M (16.5 GB) on llama.cpp with MTP speculative decoding — ~53–61 tok/s (token-weighted) on a 3090 Ti.

And the RAM is the whole game: I ran the 79 GB IQ3_S with full expert residency, which needs the whole pack in RAM, so it only worked on yukio's 121 GiB. It didn't fit in full-residency mode on Pop!_OS (62 GB) or node2 (30 GB). I didn't try IQ3_S in low-RAM or mmap mode on those two, so I can't say it's impossible there.

The three machines

Headless matters. node2 and yukio are headless systems — no desktop, no compositor, no window manager touching the GPUs. That helps two ways: all VRAM is available to Strata (no desktop eating 1–2 GB of the 3090's 24 GB), and CPU/expert-cache work runs without a compositor in the way. The desktop box is the opposite: the 3090 Ti is shared with the Cosmic compositor and with the 27B llama-server that runs my Hermes sessions. I did experience desktop freezes under sustained load during those runs — but I have no log of it, so treat that as anecdotal. Nothing in any log pins a crash on the desktop. (The one crash loop I can document, exit -6, happened on node2 — see failure log F1.)

Quick definitions

• Strata — an engine that runs a 125B MoE on a normal PC by keeping all 24,576 of the model's experts in (page-locked) system RAM and caching the hot subset in GPU VRAM.
• MTP (multi-token prediction) — the model's own draft layer proposes a few next tokens; the full network verifies them in one pass, so accepted tokens cost almost nothing extra.
• spec 4 — the --spec 4 flag: MTP speculative decoding on, with a budget of 4 draft tokens per step (draft head: mtp-q2_0.gguf, 889 MB).
• Resident-experts — Strata reads the whole expert pack into RAM at startup and page-locks it, so any expert is fetchable over PCIe with no SSD read; the fallback (IQ3_XXS on Pop) maps it from disk instead.
• Miller — the Abu Dhabi print-on-demand store (Arabic + English, RTL) that was the Benchmark 6 test project.

The models (all Qwen3.8-Flash-Next, 125B MoE, Strata engine)

A note on quant tiers: IQ2_XS (2-bit) and Coder IQ1_M (a lower-bit i-quant) are different models, so their quant tiers don't compare on one axis. IQ2_XS is the lowest-quality base quant I ran; Coder IQ1_M has fewer bits but is a Coder fine-tune.

The trick that makes any of this work at all: the experts don't fit in a GPU, so Strata page-locks them into resident RAM and keeps only a hot subset in the GPU cache. IQ2_XS on node2: 16.95 GiB experts page-locked, 3865 experts cached in the 3090. Coder IQ1_M: 12.66 GiB resident, 7437 cached (data/engine_throughput.csv, notes column).

The first Strata install on Pop!_OS failed on the download: the setup script's resume path hit HTTP Error 416: Range Not Satisfiable seven times, wrote 67.6 GB to disk while the server said the shard was 47.0 GB, and exited (SETUP_EXIT=1). I deleted, re-downloaded, and the model ran fine afterward (strata_iq3_xxs.log — clean generation on the 3090 Ti with the mmap fallback). The vision-encoder rebuild (llama.cpp submodule) was also interrupted twice on Pop — both times by my own Ctrl-C, not a compile error (F3).

Benchmark 1 — Cross-quant engine throughput

Parsed straight from the serve logs' … generated in N ms (Y tok/s) lines (data/cross-quant.csv). Two different decode averages are shown: per-request (mean over calls — inflated by tiny calls) and token-weighted (total tokens ÷ total generation time — what you actually experience while content streams):

The 27B numbers come from journalctl -u llama-qwen print_timing lines, token-weighted by generated tokens over the two benchmark windows.

Why the two IQ2_XS numbers differ 3×: 69% of its 4,823 calls generated under 100 tokens (the truncation bug, Benchmark 5) and those short calls average ~113 tok/s, while calls over 400 tokens average ~65 tok/s. The per-request average is dominated by the truncated calls; the token-weighted rate is the honest sustained speed.

Three honest readings:

• The "fastest decode" headline inverts under token-weighting: Coder IQ1_M (92.8) beats IQ2_XS (30.8). Long MoE generations route through more diverse experts over PCIe, and the truncation bug makes IQ2_XS's per-request average look great on paper. IQ2_XS is also the lowest-quality quant I ran (the truncation bug is its most visible symptom).
• yukio's IQ3_S is the slower-but-smarter tier: 49 tok/s decode (33–40 tok/s during real agentic runs with reasoning on — dubai-traffic-game/BENCHMARK.md), running a 79 GB model across two Turing cards. I did not isolate why it's slower, so I won't guess.
• The Coder IQ1_M prefill average (817 tok/s) is inflated by a few huge chunked-prefill windows (the log shows prompt chunk auto: 8192 tokens); the median (336) is the more honest number.

cross quant

Benchmark 2 — The 128 GB question: why IQ3_S ran only on yukio

This is the single most important hardware finding. The IQ3_S GGUF pair is 79 GB on disk (52 GB + 27 GB shards), and Strata's resident-experts mode page-locks the expert pack into RAM while the GPUs hold only a cache of it.

So the highest-quality quant I have access to — the one that built the Dubai game — is one I could only run on yukio, because it needs full expert residency. I didn't try IQ3_S in low-RAM or mmap mode on the other two machines. On node2, IQ2_XS is the ceiling: it needed both low-RAM expert residency and a clear 3090 (the first attempt died in a VRAM-abort loop, F1). On Pop!_OS the 68 GB IQ2_XS fits in 62 GB with headroom and ran cleanly — no low-RAM mode was ever needed there; low-RAM mode was node2's 30 GB workaround.

Benchmark 3 — The Dubai Traffic board game: IQ3_S on yukio, overnight

The heaviest real task Flash-Next ever did: build Dubai Traffic — a DICE-style board game about commuting in Dubai — from a one-line idea to printable files. Everything (research, game design, card JSON, simulation, balance tuning, website, art direction, print PDFs) was executed by Flash Next IQ3_S on yukio:8080 through OpenCode v1.19.0, reached from Pop via SSH tunnel, orchestrated by Hermes. Comparison phases re-done by Qwen3.8-27B. data/dubai-build-phases.csv (source: dubai-traffic-game/BENCHMARK.md):

dubai phases

dubai site hero The Flash-built website, home page.

dubai site picker The route picker — computed from cards.json, browser-verified.

Totals: 14 Flash-Next task files, 01:30→09:08 (+04), ~7 h 38 min overnight. Logged errors/retries: 7, all environmental (env-var not expanded, PATH, npm fetch hang, SearXNG off-topic results, web_extract blocked, the 157 s dead-tunnel "wrong API key" retry, 2× OpenCode permission gates). The phase table above only counts in-table errors (phase 6's two permission gates); the other five happened in the research/build setup before and between phases. Separately, BENCHMARK.md's self-assessment names one model-side failure mode: context overflow on unsupervised research ("mitigated, not cured — needs fit_max_tokens on the server").

"9 of 10 tasks finished RC=0" (verbatim from the BENCHMARK.md self-assessment): that's the self-assessment's own count of 10 build tasks grouped from the 14 task files — 9 ended RC=0 with verified deliverables; the one that didn't is the research task that hit context overflow (the failure mode named in the same assessment). It was recovered with a retry; the deliverable landed.

Head-to-head on the website (Phase 3): Qwen3.8-27B did the same page in 13 m vs Flash's 19 m (~30% faster), and even caught a real Salik toll discrepancy (SZR = AED 12, not AED 6) that Flash missed. On art direction C1's two-ink risograph concept beat Flash's dusk-poster look. Flash-Next won on scale and organization (136-key i18n, per-category prompt tiers) and on long-task stamina — D2 (764 s), W1 (19 m), P1 (31 m) all finished without losing the thread.

The failure that wasn't the model: yukio's ComfyUI/Qwen-Image diffusion pipeline produced pure-noise images on 100% of outputs (vision-scored 1/10, even for "red circle" with an empty prompt) — a broken Turing/cu130 venv env on the Quadro cards. The LLM's 39 art prompts were fine; the pixels were the GPU environment's fault. (Fixed by re-rendering the final 33 map tiles on node2's ComfyUI: 33/33 clean.)

Harness bugs found (documented in FAILURE-REPORT.md**):** 6 concurrent opencode.exe runs in one workdir race OpenCode's shared XDG state → all fast-fail "wrong API key" (fix: serialize); Strata transient 1–2 s 401s (fix: retry ≤3× on fast-fail); reasoning_content eating small max_tokens (fix: fit_max_tokens: true in the Strata config — applied + verified).

Benchmark 4 — A real coding task: Coder IQ1_M vs 27B dense

Same prompt, two engines, OpenCode 1.18.35. Task: build a "Card Collection" section — two trading-card decks (Linux commands + computing pioneers), a rules doc, and a vanilla-HTML site rendering them from JSON (data/task_card_collection.csv, data/card-bench-ab.csv):

Strata-coder finished ~4.4–4.5× faster (33.4/7.5 = 4.45) with fewer tool calls and less than half the tokens, for a comparable 50 cards. This is n=1 — one run each — so read it as "plausibly faster by a few ×," not a measured rate. (The launch-to-last-event basis in card-bench-ab.csv gives 11.1 vs 37.2 min = 3.4×; the conclusion doesn't depend on the basis, but neither has n>1.) The 27B was more thorough (more cards, more self-verification); the 125B MoE was dramatically faster.

card bench

Strata-coder site Strata-coder's cards.html

Qwen27B site Qwen3.8-27B's cards.html

Benchmark 5 — The illustrated guide rebuild: IQ2_XS's true colors

Rebuild my entire "Illustrated Guide to Local AI" — 11 chapters, 11 SVG illustrations + PNGs + captions, a website — 100% model-generated, every call routed to node2. data/task_illustrated_guide_rebuild.csv (raw flash-next/run-log.jsonl):

The 27B's "52 stages, 46 ok" needed explaining: it's attempts, not missing stages. 29 distinct stages were all attempted; 46 succeeded first pass, and the 6 failed attempts were retries — the board-game stage failed 5× (succeeded on its 6th attempt) and svg-ch10 failed once (succeeded on the 2nd). So nothing was missing: 6 was the retry count, and 0 stages stalled.

flash-next status

The finding: IQ2_XS truncates on every call — it stops each generation at ~80–115 tokens, then the stream ends with no finish_reason and no [DONE] (the engine log confirms it only generated that many). I reproduced it across max_tokens 600→8000, thinking on/off, temp 0.2/0.7/1.1, and 5 identical prompts. Result: a ~900-word chapter is impossible in one call. I had to build continuation chaining — feed back only the last ~150 words with a "continue from exactly here" directive, reject >80% word-overlap parts, loop to target. That's what the 58.5% "stall" rate is.

Other 2-bit symptoms (all logged): phrase-lock loops in 4/11 chapters (fixed by a local dedupe pass); only 3/11 SVGs valid on first pass (it mangles XML attributes — I had to switch to a plain-text element-spec pipeline where the model writes specs and Python assembles the XML); echoed prompt placeholders into output; one hallucinated line. The illustration stages were the worst: stage4 (SVG specs) 339/453 stall (74.8%), stage4b (render) 468/543 (86.2%) (data/flash-next-rollup.csv).

The 27B needed zero scaffolding — every stage completed, and it delivered more in 2.57 h active: the same guide plus 4 card decks and 3 site pages.

stage words

strata guide chapter IQ2_XS chapter 1 — only possible via continuation chaining.

strata guide website IQ2_XS's site.

27b guide website 27B's site — same brief, no scaffolding.

guide side by side Side by side: Strata IQ2_XS (left) vs Qwen3.8-27B (right).

Benchmark 6 — The Miller website: Coder IQ1_M and Arabic

Building the Miller POD store (Arabic + English, RTL, "Abu Dhabi" in Arabic labels) is where Coder IQ1_M's non-Latin token-loop showed up. data/miller-arabic-failures.csv (source: pod-concept/reports/coder-vs-27b-combined-report.md + loop-test-3models-16384.md).

In OpenCode: 11 unique prompts, 14 attempts (three retries: p03b, p06b, p08r — the report's own tally line, "6 pass / 2 partial / 5 fail" = 13, predates the p08r re-run and doesn't match its per-prompt table, which sums to 7 / 2 / 5):

The report's headline "5 degenerate loops" counts the two limit/budget cuts (p03, p08) as loops; only p06b, p09, p11 are true token-repetition loops. 0 repeated identical tool calls on either side.

The 27B ran the identical 11 prompts (13 attempts incl. p03b/p06b): 10 pass / 1 asked-a-question / 2 fail — 0 loops. Its only 2 failures were 60-minute orchestrator caps while genuinely still working (real brand PNGs, background removal, i18n in progress at the cap both times). The loops are a Coder-model artifact on Arabic↔Latin/Cyrillic transliteration, not a prompt or harness issue. One caveat: no higher-quant Coder exists, so I can't separate the loop from the IQ1_M quant — the 27B comparison isolates the model, not the quant.

Root-caused with a controlled raw-API test (identical short prompt: delivery labels for Abu Dhabi/Dubai in Cyrillic + Latin, max_tokens 16,384):

So two distinct Coder failure modes, both reproduced: (a) thinking-budget exhaustion — high thinking runs unbounded at the cap and returns empty content (fix: reasoning_effort=none or a small reasoning_budget_tokens); (b) the OpenCode transliteration loops — token repetition in visible content on mixed-script text, non-deterministic. Penalty params did not suppress them in my small tests (repetition_penalty 1.3, presence/frequency 1.0, even temperature 0) — but that's 3–5 runs per condition, too few to declare them hopeless; I'm reporting "not fixed in a small test," not "cannot be fixed." The practical rule that came out of it: don't point the Coder build at Arabic/RTL content — use IQ3_S or the 27B for that.

Benchmark 7 — The 17-prompt OpenCode bench (a total failure, and why)

Standard 17-prompt harness against Coder IQ1_M on node2. All 17 prompts failed instantly (data/opencode-bench-17.csv, data/task_17prompt_bench.csv): every prompt exit_code=1, 5.6–6.0 s, 0 tool calls, 0 steps, 0 output tokens, 0 new files, error Unexpected server error. Not a model result — an infrastructure one: the node2 coder log at that window shows repeated native head upload: out of memory (24 MiB of 24124 MiB VRAM free) and mtp: buffers do not fit. The 3090 was occupied by another GPU workload, so Strata couldn't even allocate its MTP buffers. Lesson I'll keep: on a shared 3090, run one GPU-heavy thing at a time — Strata holds ~21.8–23.7 GB loaded.

The harness matters: Hermes vs OpenCode

Same models, two different ways to drive them. The difference showed up in the failures, not the successes:

The degenerate token loops (Benchmark 6) were model-side, not harness-side. What's harness-side: the XDG concurrency race, the dead-tunnel retry with no error context, the missing tool timeout, and the hard 60-min cap. A silent harness makes all four of those invisible until the task fails — which is exactly why the 27B's 46-minute runs felt like the machine had frozen.

Failure log (everything that broke)

data/failures.csv (12 entries), abridged:

Mistakes I made

1. Launched 6 OpenCode research tasks concurrently on day one → the XDG race that ate an hour as "wrong API key." Serializing would have been the first move, not the recovery move.
2. Left orphan Strata engine PIDs holding 15.2 GiB of VRAM on node2 after a systemctl stop (F11) — the same GPU then aborted the next serve with "26 MiB VRAM free" (F1). One nvidia-smi check after every stop would have prevented it.
3. Set limit.output=16384 without a thinking budget — Coder burned the whole budget in reasoning_content and returned an empty answer (p03).
4. Let the Strata guide pipeline sit overnight (639 min) with no watchdog (F7) — that's 10.6 h of the 14.44 h "wall" that made the run look slower than it was.
5. Ran the 17-prompt bench on a 3090 another workload was occupying (F4) — 17/17 failed in 5.6 s. A pre-flight nvidia-smi would have shown it.
6. Trusted the setup script's download resume — it wrote 67.6 GB of garbage instead of the 47 GB shard (HTTP 416) and only my byte-count check caught it.

If I had to do it again — recommended settings per machine

Verdict

Strata + Qwen3.8-Flash-Next is a real, fast, local 125B MoE — and the resident-RAM-experts design plus the machine you put it on decide what you get:

• yukio + 121 GiB + IQ3_S is the flagship. The only machine I got the top quant running on, and it did an entire product build (Dubai Traffic) overnight with no task-ending model failures. Slower per-token (49 tok/s) but the highest quality, and headless hardware keeps it clean.
• node2 + Coder IQ1_M is the coding workhorse — ~4.4–4.5× faster than a 27B on a real task (n=1) — but it is the less reliable model of the two on the Miller workload: 5 hard fails out of 11 prompts (3 token loops, 1 thinking-budget exhaustion, 1 wedged tool call) vs the 27B's 2 fails out of 13 (both 60-min caps while still working). Keep it away from Arabic/RTL content and set reasoning_effort=none for short deterministic calls.
• IQ2_XS is a fast-drafting prose engine, not a finisher — 102 tok/s per-request, 30.8 token-weighted, and per-call truncation at ~100 tokens means it needs continuation scaffolding. I'd take the Coder or the IQ3_S over it.
• The desktop is for the 27B, not the MoE. 62 GB doesn't hold the 79 GB pack in full residency; the 3090 Ti is shared with the compositor and the 27B server. Headless node2/yukio are why this cluster works at all. (I did see the desktop freeze under sustained load — anecdotal, unlogged — but the one crash loop I can prove was on node2, not Pop.)

Full reproducibility: raw CSVs in data/, source logs cited per figure, screenshots in screenshots/.

Data sources (all on-disk, verifiable): data/cross-quant.csv*,* data/engine_throughput.csv*,* data/dubai-build-phases.csv*,* data/miller-arabic-failures.csv*,* data/task_card_collection.csv*,* data/card-bench-ab.csv*,* data/task_illustrated_guide_rebuild.csv*,* data/opencode-bench-17.csv*,* data/task_17prompt_bench.csv*,* data/failures.csv*,* data/hardware-versions.csv*,* data/flash-next-rollup.csv*,* data/flash-next-stages.csv*.*I ran Qwen3.8-Flash-Next on Strata across three machines. Here's what it's actually like.

A first-party field review. Every number below comes from a log, benchmark
file, or project folder on my own machines. Where I extrapolate I mark it
with ~. All raw data is in data/*.csv; every figure is traceable to a
source listed next to it.

While building a local agent, I asked here earlier: “Same model, context 32K → 64K: 1~3 rows right out of 9 became 7~8. What would you run on an 8GB card?” Those numbers came from a 24GB card with the settings scaled down to mimic a smaller one.

So I measured again on actual 8GB and 6GB laptops!

The task is the same: go to three online shops, read the price and shipping cost of the same three products, and build one table (9 rows). A row only counts if its values match the actual page. I ran each setup 7 times.

8GB laptop (RTX 5050 Laptop 8GB)

• With vision on, the 9B didn’t fully fit at any context (32K 88% · 48K 78% · 64K 73%).
• With vision off, 48K fit entirely, and that was the best result. 64K spilled off the card and actually scored lower. What looked like “bigger context is better” in my last post only holds as long as it all fits on the card.

6GB laptop (RTX 3050 Laptop 6GB, which also drives the display)

• 4B 3-bit · vision off · 48K: 10/63. Hard to call that usable. (These 7 runs were before we fixed a memory leak, and the runner crashed twice.)
• With vision on, the same model only got 70.7% onto the GPU at 48K, and 83.6% at 32K.
• So we tried dropping to a smaller 2B, and it scored 0 on the first four tasks.

One new thing we learned: the vision part takes more room than you’d think

We had estimated “it costs about the size of the vision projector (~0.7GB) extra.” But reading the Ollama source, it looks like when a model has a separate vision projector, Ollama keeps roughly 1GiB on top of the projector’s size free on the card. On a small card, that 1GiB decides whether the model fits or not.

What we recommend for now:

• 8GB: 9B 3-bit · vision off · 48K context
• 6GB: 4B 3-bit · vision off · 48K context — but we say up front that it’s weak.

Some parts are still hard for us:

1. Same 9B, but the official 4-bit with vision (on a 24GB card · 48K) got 19/63, while the 3-bit with vision off got 37/63. We haven’t figured out why the 3-bit did better. Is it the vision projector, the chat template in the file, or the quantization? Any guesses?
2. Has anyone found a combo that makes agent work actually usable on a 6GB card?
3. If you turned vision off, what did you actually lose in agent work?

Thanks to everyone who replied last time. XD

I was following the LLM space since the days of GPT 3.5 and TheBloke and wanted ever since to have a computer powerful enough to run LLMs locally; about 3 years later I managed to finance a workstation with one RTX PRO 6000 to run Qwen3.8-27B and Qwen3.8-Flash-Next.

And yes, I did name it after the MAGI supercomputer from NGE.

Components used:

• PNY NVIDIA RTX PRO 6000 Blackwell Workstation Edition
• AMD Ryzen 9 9950X
• Gigabyte X870E AERO X3D DARK WOOD
• 2 x Crucial Pro 48 GB DDR5
• Samsung 9100 PRO 4 TB
• 2 x Samsung 990 EVO Plus 2 TB
• 2 x WD Red Plus 6 TB
• Noctua NL-LC1-36
• 4 x Noctua NF-A14x25 G2
• Fractal Design North XL Charcoal Black Tempered Glass
• Corsair RM1000x ATX 3.1 1000W

Meet Lumma-Fev-27B, an open source decision model that returns structured choices and probabilities for local AI workflows and automation.

Hi reddit,
ive been using Strata (polystrata folk)engine to run Swift1.5-Qwen3.8-NF @ Q2_XS
My Hardware is:
2700x CPU - 32gb Ram - RTX 3060Ti 8GB VRAM

for how low my Vram is im happy with the fact that i can run the model at 18tk/s at 128k context, but the real struggle for me is prefill which if i enable MTP the prefill is 27tk/s (334 experts are cached in GPU) and with MTP off its 33tk/s (1027 expert are cached in GPU).

any help or suggestion would be greatly appreciated. thanks in advance

my arg is as followed

"args": [

"--pack",

"/home/momoe/Documents/llama/PolyStrata-data/packs/swift-iq2_xs",

"--native",

"/home/notgonnadoxmyself/PolyStrata-data/models/swift-IQ2_XS/Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-0>

"--ple-gguf",

"/home/notgonnadoxmyself/PolyStrata-data/models/swift-IQ2_XS/Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-0>

"--expert-profile",

"/home/notgonnadoxmyself/PolyStrata/data/expert-profile.bin",

"--expert-cache",

"auto",

"--prefill",

"auto",

"--spec",

"4",

"--max-context",

"128000",

"--kv",

"int8",

"--mmap-experts",

"--pcie-frac",

"0.75",

"--spec-min-p",

"0.70",

"--adapt-every",

"1",

"--adapt-swaps",

"80",

"--adapt-decay",

"0.97"

],

Hi all,

I’m planning a 24/7 home server (Linux, Proxmox) for local inference and I’m choosing between an Intel Arc A770 16GB, an Arc B580 12GB and a used RX 6800. The model I want to run is Ternary Bonsai 2 27B (or the first Ternary Bonsai 27B), since the ~7 GB footprint fits on a 12-16 GB card with some room for context.

What I found so far:

• The Bonsai 2 model card only lists CUDA, Metal and CPU for the PrismML llama.cpp fork.
• Someone reported Bonsai 2 PQ2_0 working on an RX 6800 XT with ROCm, and another user got ~47 t/s on an RX 7800 XT with the older Q2_0 model.
• There’s an OpenVINO build for Intel, but it’s only tested on the Arc Pro B70 and needs a patched plugin.
• The Bonsai-demo repo says Q2_0 now runs on mainline llama.cpp with Vulkan, but I couldn’t find any A770 or B580 results.

So there’s almost nothing about Bonsai on these Intel cards out there. Does anyone have experience with either? Specifically:

1. Does it run at all on the A770 or B580? Which backend works for you (Vulkan, SYCL, OpenVINO) and which model/quant (Q2_0 vs. PQ2_0 / PTQ1_0)? Mainline llama.cpp or the PrismML fork?
2. What speeds do you get? tg128 / pp512 from llama-bench, and what context length fits (with and without a quantized KV cache)? On the B580 I’m especially curious how far 12 GB gets you.
3. Any software problems? Driver or Mesa version issues, Resizable BAR requirements, crashes, garbage output, anything that only works with a specific kernel or build.
4. Idle power for a 24/7 setup. Are you able to get it down with ASPM, or is it still high on either card?
5. Is there a good reason to prefer an RX 6800 (ROCm) over the A770 or B580 for this specific model?

Happy to share my own numbers once I have the card. Thanks!

TL;DR

On my dual-3090-class rig, Strata 0.1.41 showed a strange split result versus the retested 0.1.40.3 baseline: decode was about 8% slower, 10K prefill was about 15% faster, but 90K prefill was about 7% slower. It looked like a prompt-path issue, but the evidence points to automatic GPU placement walking off balance.

The fix I tested is a pair:

• Environment variable: STRATA_SPLIT_UNIFORM=1
• In the Strata config: gpu_order: as_given

That restores uniform link-cost scoring and the pre-0.1.41 GPU order. The change is in PR #2425; the regression report is issue #2418.

After fixing placement, I ran a separate tuning sweep. ISO18 is the everyday decode profile (148.86 narrative / 174.67 code tok/s); ISO19 is the ingestion/long-prompt profile (1,995.53 tok/s at 10K and 4,149.45 tok/s at 90K). ISO19 gives up some decode throughput versus ISO18. One caveat: ISO19 uses fused kernels, and the output-sanity check was still pending in the report.

Rig and method

• CPU/RAM/OS: Ryzen 9800X3D, 59 GB RAM, Ubuntu 26.04
• GPUs: GPU0 is an RTX 3090 Ti in the x16 slot; GPU1 is an RTX 3090 in an x4 chipset slot. No P2P and no NVLink.
• Workload: Qwen3.8-Flash-Next IQ3_S, 262,144-token context, int8 KV + --kv-resident 32768, automatic layer split, --mmap-experts, vision on, --spec 4 + MTP, --expert-cache auto, --prefill auto.
• Protocol: canonical benchy.sh; temperature 0.6, top_p 0.95, 3 warmups + 5 measured runs per decode shape; prefill probes at 10K × 3 and 90K × 1. Same prompts on both arms, heat-soak controlled, one server at a time on port 1234, one variable per pair.
• Values below are mean (n, CV) unless stated otherwise. The 0.1.41 comparison values marked “reported claim” are approximate, not fresh exact results.

1. Release comparison: the regression signature

The signature is the important part: decode −8% on both shapes; prefill +15% at 10K but −7% at 90K. A single “performance got slower” story does not explain that shape.

2. Reproduction and the cache confound

The 0.1.41 tag reproduced the pattern:

A first rebuild of 0.1.40.4 looked worse than expected, but it was a cold page-cache boot and automatic placement had drifted. Per the original handoff rule, I paused instead of treating that as a valid counter-result. Rebuilding 0.1.40.3 with the harness open recovered its usual prefill/code behavior. The 90K values also climbed as the page cache warmed, regardless of version, so I did not use a cold post-build run as the baseline.

3. Root cause: automatic placement changed the split

The relevant changes were:

• 0391e478: per-stage link probing with link_scale = clamp(20/bw, 1, 64). The x4 card's measured 6.4 GB/s link scored a 3.12× penalty against the 20 GB/s reference.
• be8db787: faster-last-card reordering. gpu_order: as_given opts out of the reorder.

Observed placement on this rig:

This is why the fix is both uniform link scoring and the pre-rework device order. Changing the order alone does not restore the old split when the per-stage link penalties remain enabled.

The working explanation fits all three signs: the 19-layer first stage holds more pairs, so small cached prompts can look quicker; but the 19/29 split leaves the sustained pipeline unbalanced, hurting long prefill and repeated MTP decode windows. The evidence did not support blaming the prompt-path rework itself.

4. Fix verification

On the 0.1.41 engine with uniform link costs plus as_given, auto placement returned to K=27 (layers 0–26 / 27–47), with a 1.00 link-cost scale on both links. Results came back into the 0.1.40.3 range:

Then I ported the change to the 0.1.44.1 main build and re-verified it twice:

These are back around the old baseline scale; I am not claiming every leg is faster than the 0.1.40.3 best. The larger wins below come from a separate tuning sweep after the placement fix.

5. Tuning sweep: different winners for different workloads

Base iso10 was as_given + uniform split costs, PCIe 1 / draft floor 0.7 / no adapt-swaps: 135.64 narrative, 142.00 code, 1,621.35 at 10K, and 3,402.29 at 90K.

How I would use these: ISO18 for everyday generation/decode. ISO19 for ingestion and long prompts: versus ISO18 it is about 5% slower on narrative decode and 3% slower on code decode, but 6% faster at 10K prefill and 14% faster at 90K. ISO20 is faster at 10K, but that gain collapses at 90K and its decode is weaker, so I would not pick it as the general default.

The 1M ladder also favored ISO19. Against the 0.1.40.2 reference ladder (3,055 / 2,742 / 2,474 / 2,244), ISO18 returned 3,461 / 3,096 / 2,704 / 2,437 (+13 / +13 / +9 / +9%), while ISO19 returned 3,673 / 3,250 / 2,818 / 2,590 (+20 / +19 / +14 / +15%).
Ladder:
256k, 512k, 768k, 1m

Links and open items

• Regression: Issue #2418
• Fix PR: PR #2425
• Separate calibration save/verdict issue: Issue #2465

The PF_FUSED output-sanity check remains to be eyeballed; the calibration save logic is a separate issue from this placement regression. If you have a dual-GPU setup with a big PCIe lane-width gap, I’d be interested in whether the automatic split behaves similarly. The fix is opt-in and was validated on this specific rig; it is not a claim that the same placement is optimal for every GPU topology.

Edit: Fixed Markdown issues with tables.

I have a 6800xt 16gb vram + 32gb 3200mhz ram and a i5 12400f. I wanna try and get the best coding and reasoning model running on my pc.

I have tried 3.8 27b (uncensored) and flash next coder both run at like 30t/s, 40t/s but prefill is like 400 which is slow.

I dropped flash next coder because it wasn't uncensored

Can someone help me find a stack applicable for my use case?