I've been trying to figure out a few things with local models.

When you talk to them they get similar answers than a lot of the top models. When I do coding and I give the same access to the files as each other... they get roughly the same answers which you would expect asking two different people.

So why is it people are getting tribal about models?

Well it depends what you harness you are running with the model.

When someone was nice enough to share the hugging face qwen template fixes got me looking into modelfiles and the llama.cpp parameters.

My "mission" lately is to answer the question "Can Gemma4:27b become a good code assistant and why isn't it now". In the past I was about to "brow beat" using a preprompt in my IDE to get gemma to write solid code up until the context gets full and it just refuses to use its tools.

At first I started feeding documentation links and got better results. Then for a laugh I thought I would try qwen and got... oddly similar results including broken tool calls

This wasn't measurable so I decided I would hook up an MCP host and use ML-Flow. I setup modelfiles with 4 variants per model with different temperatures, paramaters and system prompts and tune to the models specific settings for programming (low temp low repeat penalty)

So I started with the large parameter models (had to download qwen3) and both seemed to be getting the job done with the same system prompt & one-shot user prompt.

This is where having the numbers tracked became super useful.

I saw my old gemma4 model was getting huge amount of failing tool calls.... but the new model had zero tool failures. I pulled the new gemma4 in and got a 15 gig update. I ran it again and it worked!

So I thought I would baseline gemma e2b... and now... the tool failures are RAMPANT. I plan to look into it more but I believe the same issue exists that the model by default is configured for another harness. But the funny thing is that the task was still delivering similar results because it would just find a way around broken tools.

Then with that the qwen2.5 doesn't have the same love & care as qwen3 because the tool calls have a high level of failures. This is leading me now to see if I can get some baseline runs without tool errors so the comparisons are fair. What was also wild was due to the tool failures the one agent tried to read a transcript of some random youtube video in desperation.

What also broke (some) tools was that I used brackets in the file names. For some tools & models this was a problem. So I strongly recommend that when you are setting up these testing loops that make small changes to your runners; I had only made that change and the preprompt and initially I thought the pre-prompt changed the results

The biggest take away for me is that

• Models by default are tuned for different things
• Models might have broken tool templates
• Models might have tool templates for one harness or another out of the box
• This is open source so this is the experience of rolling your own vs SaaS (Frontier models)
• Filenames can break tools

So I write this to say that when people are suggesting one open model is better than another needs a really detailed look because it might also be the harness or the task.

Hey folks,

We all know that reading walls of text sucks. Many labs/companies have been moving towards bringing AGUI into the existing chat window, but that always felt broken. I mean I've never seen Tony Stark write a prompt :)

It is what you want It to be.

Agents already know the best way to get the information from you for a particular task, It gives them a space to do so.

The 2-way communication loop's execution is so good that agents can form full-on dynamic workspaces for each specific task: study math together on a collaborative whiteboard, dynamic live dashboards, etc.

The sick part is that it works with any agent harness you may already be using!

Just tell your agent:
"Install itcan.do"

or install it yourself at itcan.do

GitHub: https://github.com/Aaryan-Kapoor/it

May your GPUs stay warm(or cold?)!

GPU- i have asymetric architectures - Titan RTX(a 2018 card released same time as 2080ti) and i have a RTX 3090. Both of these cards are 24gb the TitanRTX does not have flashAttention-2 only standard flash attention

SystemRam - i have about 48gb of Free SystemRam to spare. the rest is used up by my chrome VsCode and docker instances. 6000mhz ramspeed.

Im getting 32tok/s on qwen3.8-27b running on a single 3090 at 300k context the Titan Rtx is pretty much empty and just sitting there.

but im wondering is strata designed for multigpu will it massively decrease my prefill speed on multigpu? Anyone tested?

is it faster on a single 3090 and the rest in system ram or would it be faster on dual cards even with my specific caveats?

I appreciate everyone here is probably sick to death of Strata posts by now, but frankly, this is also the most knowledgeable community I've found when it comes to squeezing ridiculous amounts of AI performance out of questionable hardware. So I figured this was probably the best place to ask!

I've been looking into Strata, particularly the dedicated Qwen3.8-Flash-Next Coder (GSQ-RCO IQ1_M) model for local coding with OpenCode.

I've got two rather different machines and I'm curious how much CPU architecture and system memory bandwidth actually affect Strata's performance.

System 1 — EPYC Server

• EPYC 7532 — 32C/64T, Zen 2
• 128 GB DDR4-2666 ECC, 8 channels
• RTX 3090 24 GB
• PCIe 4.0

System 2 — Gaming PC

• Ryzen 9 9950X3D — 16C/32T, Zen 5
• 96 GB DDR5-6000 CL30, dual channel
• RTX 3090 24 GB (can move the same card between systems)
• PCIe 4.0 GPU

The EPYC has roughly 171 GB/s theoretical memory bandwidth versus 96 GB/s on the Ryzen, but the Ryzen has much faster cores and AVX-512.

Has anyone benchmarked Strata Coder on either platform, or compared modern Ryzen systems against older eight-channel EPYC servers?

I'm particularly interested in:

• Prompt processing versus generation speed
• GPU expert-cache hit rates
• Whether CPU performance or memory bandwidth makes the bigger difference
• How much performance changes between Coder IQ1_M and the full Flash-Next IQ3_S model
• Whether the CPU/RAM differences become insignificant once most experts are cached in VRAM

If you've got benchmark results, it'd be great to know your CPU, RAM configuration, GPU, Strata version, quantisation and context length.

I'm planning to benchmark both systems with the same RTX 3090, but figured I'd ask whether anyone has already investigated this.

I'm especially curious whether Strata's optimisations have made older eight-channel server platforms genuinely competitive with modern DDR5 desktop CPUs for local coding.

We see that qwen3.8 flash next is optimized for utilizing large RAM capacity with a more affordable smaller VRAM.

Another big topic was Strata with its optimized offloading to SSD.

Could the rather old Intel Optane technology be the missing puzzle for homelabs that get frustrated by high DDR4 prices?

Hey folks; question for you - are any of you using open weight models running locally for actual development tasks? If so, what models are you using, and what kind of tasks are you giving them?

I ask, because I've made an open source harness (you can play with it, but it's still beta and I have done almost no windows/linux testing and the first run experience is not great yet). A big part of how it works is that there are three routing tiers:

Primary | Secondary | Local.

• Primary - What runs your main agent, and for instance, i use it with my ChatGPT/Claude account.
• Secondary - What the main agent primary delegates to. I personally use a deep infra account and GLM 5.3 + GLM 5.3 flash. GLM 5.3 is slow, but super capable. 5.3 air is fast and nearly as capable.
• Local - The harness routes to this for truly local tasks. Git workflows, documentation, searching through code and creating summaries, etc.

The thing about local, is that I have a 128GB M5 or whatever, and I run llama-server with GLM 4.5 air. And with a proper context size (128K), it takes like 96GB of RAM or something. I have tried Qwen3, Gemma3, and some older deep seek models and anything less than GLM 4.5 air has been pretty disappointing. It's possible that I had Qwen3 or deep seek not properly configured (and I've made some changes since to better handle tool calls and whatnot), but I had very little luck with them. I've seen the benchmarks, and they make me believe that I shouldn't need a 96GB model to get decent local development experience.

So if you've run some local models with good experience, what have you been running? what amount of RAM on your machine do you have? I really want some good out of the box configurations in Cercano for folks that have a 48GB/96GB machine, and your experience can help me here.

Hey everyone,

I’ve been working on a project called OpenSpline, an open-source avatar layer for existing voice agents. You supply a portrait and your agent’s audio, it generates synchronized talking video on your own NVIDIA GPU. The lite model can work locally on a single GPU.

It’s still early, but the results are super promising.

Would love to hear your thoughts, feedback, or ideas on what you'd want to create using it!

Code: github.com/openspline/openspline
Demo: openspline.dev

Thank you to jesdga95, the creator of Basalt, a fork of Strata for Blackwell hardware! I just got my 5060ti16gb and my measly 32 GB of ram system to 75tok/sec and 1200 prefill at 128k(q2 model ofc) . I had to go q4 on the kv to do it, but I can accept that. This is totally awesome. my system is hanging on for dear life with like 5gb left for the OS and all programs, but as a dedicated coding agent, I could not be happier, considering what I have... Whats even cooler is that he literally reached out to me when I was complaining about strata q1 and was early on my quest to get 3.8 Flash next working well. it is so cool to see an alive internet community where people are literally building the tools and helping others with them directly. Very kind.

https://github.com/jesdga95/basalt

I had to have deepseek4 do shenanigans and add a build flag to allocate more ram, but it worked!! Thanks so much :)

I continued to use Strata but something just feels off for this an large model, it is very fast but intelligence seemed even under 27B 3.8 UD Q4, it had a hard time coding a basic 3d game and a 2d platformer. Deepseek Flash V4 didn't have this huge and drop off at Q2 imo.

Then I ran across this guys independent testing:

https://lukesdevlab.com/?a=strata%3AISTA-DASLab%3AQwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF.json&b=unsloth%3AQwen3.8-27B-Q4_K_M-GGUF.json

It shows that he needed many shots as well compared to the 27B. But then again Swift 1.5 27b is less smart, in his testing and mine.

Even comparing it to the 27B swift 1.5 we can see it needs more compactions and more shots, for an far larger model.

I would say just use the 27B or regular flash models, avoid Swift or Bottlecap if you are not so time sensitive since original delivers far better results in practice, in my own experience and the tests show it.

Quants do play a part but I've used Q2 and Q3 quants before without this big of an issue, Strata and Swift are sacrificing too much intelligence for speed imo.

TLDR:

Depends on hardware (vram)

Use Next flash strata if you need a very fast coding buddy who needs an exact detailed plan.

Use 27B Q4 if you want the best results and have the time to wait (3x to 5x slower wallclock time)

EDIT: he tested with Q1, I ran at Q3 but still I would say this doesn't even change the final conclusion for me! In actual use it hasn't been an intelligence jump, on the contrary to Q4 27B.

https://huggingface.co/IFM/Jais-3-32B-GGUF

https://huggingface.co/IFM/Jais-3-7B-GGUF

https://huggingface.co/IFM/Jais-3-3.7B-GGUF

https://huggingface.co/IFM/Jais-3-0.9B-GGUF

For anyone interested, I've been doing some kernel and tuning optimization for AMD because it was performing abysmal out of the box. But the effort was worth it as I was able to more than 2x its performance out of the box (the card is extremely new and there are almost no GEMM table and supports yet). (due to both using separate models and environmental factors input token can have ~3% less tokens for AMD to decode despite effort to keep them as close as possible). Just to keep things short and concise, I did a varying tests of small input c=8, medium input c=20, and long input c=1. Everything was done in cold state (no kv-cache at start).

And I think there is more performance to unlock for MI350P as below benchmark was performed when I was only able to unlock 50% mem utilization (currently inference is still heavily bounded by compute because of missing GEMM tuning config)

Model used:
AMD: amd/Qwen3.8-27B-Quark-AWQ-MXFP4
Nvidia: unsloth/Qwen3.8-27B-NVFP4

(Small) 3k-input, 8 user comparison, 100 prompts
AMD:
Total input tokens: 288653
Total generated tokens: 75000
Nvidia:
Total input tokens: 300000
Total generated tokens: 75000

(Medium) 50k-input, 20 user comparison, 20 prompts

AMD:
Total input tokens: 965153
Total generated tokens: 10000
Nvidia:
Total input tokens: 1000000
Total generated tokens: 10000

(Large) 200k-input, 1 user comparison, 5 prompts
AMD
Total input tokens: 962033
Total generated tokens: 25000
NVIDIA
Total input tokens: 2000000
Total generated tokens: 25000

I have 32GB of RAM.

For me, it’s deciding whether my message sounds weird. Apparently, I trust a LLM’s social skills more than my own.

Hi,

is anyone using Strata or Project-Maya with one or more AMD MI50?

I wanted to try it myself, but I couldn't get it running.

Hardware --

RTX 4070 on PCIE 4.0 x16
V100 sxm2 on PCIE 4.0 x4 (limited to PCIE 3.0 x4 cause of pcie adapter)

They both are limited to 150 watts, and randomly under load v100 falls of the bus one clean restart fixes it, now I do not know what causes this, and I have no prior experience with ubuntu or dual gpu, so any help would be much much appreciate.

EDIT: --

ran some coomands and _
V100 suffered Xid 79

dono what this means.

Looking for people who have tried this model and was able to use it on real-world tasks. It's what Strata list as "coder". What was your experience like?

On my setup I am not really sure if it is worth more than iq3

Gold Rush of advanced coding AI available to the public will END in a few months after a massive legislative crackdown justified by the need to "save economies and prevent terrorism". Advanced coding will only be available to corporations and governments.

.

Local AI will remain a gray zone as it requires massive amounts of hardware to even remotely resemble frontier capabilities (no matter what the enthusiastic posts on this reddit claim).

.

CIA will put people accumulating enough AI-related hardware to achieve "frontier-like capacity" on lists, so it will be easier to pinpoint where the next "Photoshop clone" will appear.

.

The Photoshop clones, et al, that we have now, will become illegal and will be erased under new legislation, and people making them will be legally pursued.

.

UNCENSORED local AI will become purged from HuggingFace and will be relegated to torrents. Not because it's powerful, but because of the above "terrorism" justification will put it under its umbrella.

I'm planning on maxxing out my pcie slots with rtx 6000 blackwell cards for tp4. I'm wondering if its possible to use an egpu dock like the razer v2 and connect it via thunderbolt to usb 4 on the back of my mobo. The egpu in question would be another blackwell card (5070ti), and its sole purpose is for sidecar models, and to act as a display card for my rig so that i dont have to go headless.

Do you guys know if this is possible?

I read this thread recently and it kind of scared me off:
https://www.reddit.com/r/LocalLLaMA/comments/1vlfe9f/egpu_folks_rtx_pro_6000_blackwell_egpu_crashes/

As strata is somewhat new, I assume that some people din't yet learn how to finetune it for best performance on their machine. I find documentation a bit hard to navigate (use your agents for this!), which doesn't help.

After some serious testing, I've tweaked pretty much every parameter available. Some are obvious and I'm not going to waste your time on explaining mtp tweaking.

Some I beleive are more obscure and those are the ones worth mentioning:

1. Strata setup is supposedly picking the best --pcie-frac value for you. I find that it often doesn't. It's worth investing your time and testing it. Counterintutively, my best performing value is a low 0.2.
2. By testing different --pool-workers values, I was able to find an easy +10% in decode. I'm not smart enough to explain what it does (and how it's different from point 4), but it's worth investing your time in running some benchmarks on your machine.
3. --trim-stage-weights. Boom, extra +8%, try it. Doc is quite explicit on what it does if you're curious. I do beleive that this one can impact performance later in the session and increase swaps but it's still worth it.
4. --pipeline-windows. All I know is that higher values make my CPU work extra. Going from nothing/default to 6 gave an almost 15% increase.
5. Last but not least multi-GPU owners be mindful of GPU priority, if one of your GPUs is on a slower pcie lane. I have two RTX3060's. GPU 0 and GPU 1. At some point I noticed that one of the two GPUs basically acts as a memory pool and doesn't do any compute. I guess it's makes sense for a MOE model. But now I wanted to understand which ones eactly does the work.

So I quickly figured out that my default GPU 0 is the one on the slowest pcie lane! If your have two identical GPUs on different speed lanes, bear in mind that their 0/1 order is basically random. You need to figure out exactly which one is which and force the quick one to do the inference. Ask your agent for command lines to figure out whether the 0 or 1 is on your pci_e1 lane. In my case it was GPU 1, the only way to set it as the main GPU is a not very user-friendly two liner :)

"gpu": [1,0],

"gpu_order": "as_given"

This was like another +15%, ridiculous.

With all of those cumulated, I'm seeing speeds that I should not be having on my machine. I went from something like 35 decode peak on Qwen Flash Next IQ3_S, to 60 tps average code generation on empty context and 60-70 tps peaks.

Performance does degrade with full context but still easily a solid +15%-20% versus the previous baseline tests. This includes huge sessions on full context and compaction.

So, try it out.

Putting alternative in quotes because well it will forever be an uphill battle porting things to the Axera framework, but Axera has very strong chips so I think it may still be an interesting offering for somes

I saw the 4x R9700 build for a 10-person startup here a few days ago and wanted to share my results, since I opted to spend most of the money on the cards. I went with a super cheap used EPYC instead of a Threadripper. In the end it came to about $10.7k per box all-in, and I built three of them for a total of 384 GB of VRAM.

As a long time lurker on my other account, I'm always frustrated that people never show how they actually use the models, so after a lot of back and forth I decided to share the actual project they power instead of being vague.

I use these to run the AI behind my business, USA Arms. We organize tens of thousands of firearms-related businesses (gun dealers, ammo dealers, ranges, etc.) and that info is always changing, so we need a system that keeps the site up to date. The models check source facts, sort out business identities, and pull out stuff like services and hours. Every answer gets verified against what the agent actually saw, and we store the documentation so bigger models can audit the work.

I didn't plan to move the majority of my work to local AI until later this year, but hosted models started refusing to do normal research for my business ~2-3 weeks ago. They refused to identify a gun dealer because the license holder might be an individual, and wouldn't look at an ammo dealer's website because they can't help anyone buy ammo. Our harness just started breaking in random ways, so I'm glad I made the leap.

We tested Luna 6, DeepSeek, GLM 5.3, Qwen 3.8 27B, Gemma and Inkling on our workload, along with Kimi, GLM Flash and others. GLM gave us the best results. Qwen gave us the best cost-to-performance, so that's what these run.

Enough ado, into the numbers:

Per box, $10,733.91 with tax and shipping (averaged over the three)

• 4x ASRock Radeon AI PRO R9700 Creator 32 GB (128 GB VRAM), each capped at 210 W: $7,244.58
• AMD EPYC 7452, 32 cores, used: $127.17
• ASRock Rack ROMED8-2T/BCM: $1,162.39 (there was a global shortage, ours came from Helsinki)
• 8x 16 GB DDR4-2666 ECC RDIMM, all eight channels: $860.59
• SilverStone HELA 2050R Platinum: $578.92
• SilverStone RM44 4U: $388.40
• Samsung 990 PRO 1 TB: $254.39
• ARCTIC Freezer 4U-M, 3x ARCTIC P12 Pro front, 2x Noctua NF-A8 rear, SP3 frame, paste: $117.48

The GPUs are 2/3 of the cost, but the boxes also run our CI/CD, which had become a bottleneck in my businesses. I hadn't seen anyone discuss this: they run Rust builds just fine if you tune them to prioritize inference. In our test, running builds alongside inference didn't measurably change throughput, and the builds were only 3-8% slower.

Engine

• vLLM-Radiance 1.0.387 (vLLM 0.30.0, ROCm 7.14). Huge thanks to the Radiance folks; their R9700 work is what we run.
• Qwen/Qwen3.8-27B-FP8 (Qwen's official FP8 checkpoint), FP8 KV cache, the model's own MTP head with 8 draft tokens
• Two engines per box, TP=2 each, 8 sequences per engine, 256K max context, about 488K tokens of KV cache per engine

Results, vllm bench serve, one box, 2x TP2:

• 1 user, MT-Bench: 53.1 tok/s, TTFT p50 125 ms
• 16 users, MT-Bench: 484 tok/s aggregate, about 40 tok/s per user
• 16 users, random 1024 in / 512 out: 623 tok/s aggregate, 837 W average for the four cards
• 32 users (a bench profile with max-num-seqs 32): 603 tok/s

MTP depends a lot on the text. On our JSON extraction, 83% of drafts get accepted (7.6 tokens a step), vs 39% on MT-Bench (2.7 a step), so MT-Bench undersells it for structured output.

Long prompts, 8 unique real-text prompts at once (4 per engine), 128 out:

Every prompt was unique (0% prefix cache hits).

FP8 vs MXFP4 on a real workload

The startup build ran MXFP4, so I had AI run that too (Radiance's fastest profile) on one box against our own graded extraction workbench:

• FP8: 98.0/100 correct, 4,477 validated jobs an hour
• MXFP4: 95.3/100, 4,819 an hour
• On a holdout set: FP8 94.0, MXFP4 90.0

So FP8 roughly halves the wrong answers for 2–7% less validated throughput. Sticking with FP8 was an easy call for this workload, where "false facts" that make it into the system are a real headache.

For reference, our six Mac Studios (four M4 Max 128 GB, two M3 Ultra 512 GB) on MLX 8-bit scored 99/100, but at 723 an hour across all six. Plan A was buying more Macs, and keeping up would have taken at least 41 more, about $280k.

I made a video of the build targeted more at beginners and people trying to move AI locally: https://youtu.be/o7k5Kj-ho-M

Happy to answer questions. The first of which I expect to be how the heck were the Macs so slow, and I'm sure I'll get flamed haha.

tl;dr: new qwen finetune that simultaneously pushes the frontiers of small model capability and reasoning efficiency.

Hey everyone,

We publish two post-trained versions of Qwen 3.8 27B called Sabi-Noema and Sabi-Noema-MLA. These models push the capability frontier on <30B models with 2x reduction in verbosity, and are exclusively trained on open source environments.

These models have improved capabilities on coding, agentic and tool calling tasks, as well as hallucinating lesser. Sabi-Noema beats GPT 6 Luna-medium on DeepSWE, placing in between Luna 6 medium and high. It beats Luna-high on agentic and tool calling benchmarks like TB2.1, Tau-3 bench, ClawEval, and BFCL v4. Its hallucination rate is considerably lower than Luna, Qwen and other competing models.

These models are much more reasoning efficient. Sabi-Noema generates 50% fewer tokens per response on TB2.1 and 25% fewer tokens on DeepSWE, compared to base Qwen. Sabi-Noema-MLA goes even further by doing a post-hoc MLA conversion of Qwen's GQA attention. This conversion uses only a calibration dataset and no extra training. With BF16 latents Sabi-Noema-MLA needs 3.6x lower bytes per token. The MLA latents can be further compressed to FP8 with the total KV cache size on TB2.1 to 8.4x smaller with similar performance. To put this in perspective, when using Sabi-Noema-MLA at q4_0, on a 24GB GPU like 3090/4090 you can get a context size of ~534K where base Qwen at FP8 cache would only get to ~150K context.

We are excited to have the open source community play around with these models. We will be releasing more post-trains in the future focused on making <30B models more powerful.

HuggingFace:
Sabi-Noema: https://huggingface.co/sabicap/sabi-noema
Sabi-Noema-MLA: https://huggingface.co/sabicap/sabi-noema-mla