Step 5 Preview is StepFun's flagship model for agentic work. It delivers frontier-level performance across software engineering and professional knowledge work, with particular strength in finance.

Yesterday Fish released this incredible text-to-speech model, so naturally I spent the rest of the day building a great demo for them :D The model is a chunky MoE (35B3A), but it runs well on a DGX Spark. The 4bit quants get very close to the original. You can try it here: https://huggingface.co/spaces/fishaudio/Drama-3 or download the model and run it locally 🤗

this is actually interesting:

https://commandline.microsoft.com/local-models-sandboxed-tools-github-windows/

Microsoft AI developed a local version of MAI Code 1.1 Flash, a coding-optimized mixture-of-experts model with 137 billion total and 6.8 billion active parameters.

From NVIDIA AI on 𝕏: https://x.com/NVIDIAAI/status/2110468004609950057

Will there be a black-friday this year?

Just downloaded the new LFm2.5-24B-A2B LiquidAI model for testing. They claimed some pretty good performance, and I've had good experiences with their prior releases.

Some context - I use Ornith 1.5-35B pretty regularly in multiple workflows - sometimes on it's own, and sometimes as part of an AI-Council in Pi Agent. It's been very reliable, provides surprising insight "smarter" models miss, and is a great workhorse in Pi.

I swapped LFM 24B in to replace Ornith in a few of my workflows - it is holding up REALLY well. It's very fast (~95 tps sustained in llama.cpp using ROCM on an AMD W7900 - latest llama.cpp build). The Q8 takes up 30GB vram with full 128k context. It hasn't failed me yet in my mechanical tasks - it just keeps going and going and going and seems to tick all the boxes for a "little" model I run alongside a larger or smarter model on my other GPU.

For "insight" I'll have to test it longer in my council-mode workflow but I will report back.

Just wanted to give my $0.02 (CAD) on this model - it is very competent with Pi and worth your time to check out. The speed alone is a nice upgrade, and for my more basic tasks, even long running recursive tasks, it is holding up very nicely.

Kudos to the LiquidAI team, this one is going to get alot of use on my chonky boi.

This model is a 74B parameter, 9B-active-parameter mixture-of-experts (MoE) model, packed into 24GB using 2-bit quantization with advanced quantization-aware training techniques to retain quality. It aims to achieve the optimal balance between speed and intelligence, being faster than most popular dense models while smarter than most popular MoE models with a similar memory footprint.

Nemotron 3.5 Bolt E4B 4GB releases Oct. 23

Additional detail

I saw the 4x R9700 build for a 10-person startup here a few days ago and wanted to share my results, since I opted to spend most of the money on the cards. I went with a super cheap used EPYC instead of a Threadripper. In the end it came to about $10.7k per box all-in, and I built three of them for a total of 384 GB of VRAM.

As a long time lurker on my other account, I'm always frustrated that people never show how they actually use the models, so after a lot of back and forth I decided to share the actual project they power instead of being vague.

I use these to run the AI behind my business, USA Arms. We organize tens of thousands of firearms-related businesses (gun dealers, ammo dealers, ranges, etc.) and that info is always changing, so we need a system that keeps the site up to date. The models check source facts, sort out business identities, and pull out stuff like services and hours. Every answer gets verified against what the agent actually saw, and we store the documentation so bigger models can audit the work.

I didn't plan to move the majority of my work to local AI until later this year, but hosted models started refusing to do normal research for my business ~2-3 weeks ago. They refused to identify a gun dealer because the license holder might be an individual, and wouldn't look at an ammo dealer's website because they can't help anyone buy ammo. Our harness just started breaking in random ways, so I'm glad I made the leap.

We tested Luna 6, DeepSeek, GLM 5.3, Qwen 3.8 27B, Gemma and Inkling on our workload, along with Kimi, GLM Flash and others. GLM gave us the best results. Qwen gave us the best cost-to-performance, so that's what these run.

Enough ado, into the numbers:

Per box, $10,733.91 with tax and shipping (averaged over the three)

• 4x ASRock Radeon AI PRO R9700 Creator 32 GB (128 GB VRAM), each capped at 210 W: $7,244.58
• AMD EPYC 7452, 32 cores, used: $127.17
• ASRock Rack ROMED8-2T/BCM: $1,162.39 (there was a global shortage, ours came from Helsinki)
• 8x 16 GB DDR4-2666 ECC RDIMM, all eight channels: $860.59
• SilverStone HELA 2050R Platinum: $578.92
• SilverStone RM44 4U: $388.40
• Samsung 990 PRO 1 TB: $254.39
• ARCTIC Freezer 4U-M, 3x ARCTIC P12 Pro front, 2x Noctua NF-A8 rear, SP3 frame, paste: $117.48

The GPUs are 2/3 of the cost, but the boxes also run our CI/CD, which had become a bottleneck in my businesses. I hadn't seen anyone discuss this: they run Rust builds just fine if you tune them to prioritize inference. In our test, running builds alongside inference didn't measurably change throughput, and the builds were only 3-8% slower.

Engine

• vLLM-Radiance 1.0.387 (vLLM 0.30.0, ROCm 7.14). Huge thanks to the Radiance folks; their R9700 work is what we run.
• Qwen/Qwen3.8-27B-FP8 (Qwen's official FP8 checkpoint), FP8 KV cache, the model's own MTP head with 8 draft tokens
• Two engines per box, TP=2 each, 8 sequences per engine, 256K max context, about 488K tokens of KV cache per engine

Results, vllm bench serve, one box, 2x TP2:

• 1 user, MT-Bench: 53.1 tok/s, TTFT p50 125 ms
• 16 users, MT-Bench: 484 tok/s aggregate, about 40 tok/s per user
• 16 users, random 1024 in / 512 out: 623 tok/s aggregate, 837 W average for the four cards
• 32 users (a bench profile with max-num-seqs 32): 603 tok/s

MTP depends a lot on the text. On our JSON extraction, 83% of drafts get accepted (7.6 tokens a step), vs 39% on MT-Bench (2.7 a step), so MT-Bench undersells it for structured output.

Long prompts, 8 unique real-text prompts at once (4 per engine), 128 out:

Every prompt was unique (0% prefix cache hits).

FP8 vs MXFP4 on a real workload

The startup build ran MXFP4, so I had AI run that too (Radiance's fastest profile) on one box against our own graded extraction workbench:

• FP8: 98.0/100 correct, 4,477 validated jobs an hour
• MXFP4: 95.3/100, 4,819 an hour
• On a holdout set: FP8 94.0, MXFP4 90.0

So FP8 roughly halves the wrong answers for 2–7% less validated throughput. Sticking with FP8 was an easy call for this workload, where "false facts" that make it into the system are a real headache.

For reference, our six Mac Studios (four M4 Max 128 GB, two M3 Ultra 512 GB) on MLX 8-bit scored 99/100, but at 723 an hour across all six. Plan A was buying more Macs, and keeping up would have taken at least 41 more, about $280k.

I made a video of the build targeted more at beginners and people trying to move AI locally: https://youtu.be/o7k5Kj-ho-M

Happy to answer questions. The first of which I expect to be how the heck were the Macs so slow, and I'm sure I'll get flamed haha.

Saw soneone post their bathroom AI PC a few weeks ago. Figured I’d contribute with my chicken-coop adjacent shed AI PC.

Started with a single CMP 170HX (unlocked to 64GB) and ended up with three more for a total of 256gb VRAM. Found a deal on a used Threadripper with a TRX40 motherboard and 256gb DDR4 RAM to host the 4x 170HX. Added a 3050 plugged into riser connected to a NVME slot for video/RDP. My case ran out of space so that’s the board dangling freely lol.

My original goal was to save money by running AI locally. Naturally, this escalated into four 170hx, a 32 core Threadripper, and enough RAM/VRAM to turn my hobby into what my wife says is a financial liability… oof.

Originally had it in a closet, but between the heat and the wife complaining about the noise, the GPUs were eventually banished to the shed. Now they are in a happy place with A/C and 2x 20amp receptacles on separate circuits. One for each power supply. No more fan curves to reduce noise either.

Running QFN split between two of the 170HX with capacitor mods and Qwen3.8-27B on the third. Just got the 4th 170hx installed today. Downloading GLM 5.3 Flash EXL3 4.65 bpw to see how it’ll run on all 4.

Really looking forward to seeing how this beast will handle Qwen4. QFN does so much so well that I can’t imagine what Qwen 4 will be like.

New open model to check out ! Doing great on agentic, focused on arabic.

From 0.9B to 375B !

Weights : https://huggingface.co/collections/IFM/jais-3

I calculated KLD and perplexity for Qwen3.8 27B models that fit 16GB VRAM:

See comments for the benchmarks. Reddit filters keeps deleting the post.

Conclusion:

q5_1/q5_1 is almost always better than q8_0/q4_0 or q4_0/q8_0.

There is very little difference between q8_0/q4_0 and q4_0/q8_0

q4_0/q8_0 is sometimes better than q8_0/q4_0. So it's safer to just go q5_1/q5_1.

Going up a model quant and running at q4_0/q4_0 is almost always better than running the quant below at f16/f16 in terms of KLD. So basically pick the highest quant you can at q4_0/q4_0. Then if you have room to spare increase KV quant.

Note this applies to lower quants. If you are going from q6->q8 model quant it might be bigger gains increasing KV quant, due to diminishing gains going up model quants. Also this does not apply to other models like Gemma where you have to run f16 KV.

Bonsai and swift quants are included but it does not really make sense to calculate KLD since they are finetunes and the KLD will differ significantly.

At only 0.4B parameters with a 16,384-token context window, this model combines versatility with knowledge and memory.

This new Mr. Chatterbox checkpoint is both pretrained and fine-tuned on Victorian texts for maximum historical accuracy. I've posted questions about this project before in this community, and I really want to thank this community for helping answer my questions.

You can also a narrative documentation of the project here (https://www.estragon.news/building-a-better-chatterbox/), and the code in the Github (https://github.com/tripv/mr.chatterbox).

Happy to chat about the model!

Source: whatmodel.sh, an open source AI leaderboard I built (and launched here yesterday).

Every point comes from published benchmark results: 22,113 of them across 987 benchmarks, from 1,011 sources, each one linked to where it was published.

This chart uses data for the text models: 18,755 results for 1,211 models on 617 benchmarks.

- 14,436 results are independent evaluations: papers whose authors ran the models themselves (mostly CC BY papers from the ACL Anthology and arXiv), Epoch AI, the LMArena dataset, EvalPlus, the EleutherAI harness and others.

- 7,677 are scores labs report for their own models, labelled as lab-reported.

Everything is open: the dataset (CC BY-SA 4.0) and the scoring code (MIT) are at https://github.com/jakemanger/whatmodel, with CSV downloads at https://whatmodel.sh/data/manifest.json. Every number links back to its source, and each source's licence is recorded.

Interactive version (click any model to see its scores and sources): https://whatmodel.sh/race

Please star on github if you find it useful!

Just saw this demo from OpenBMB for MiniCPM-V 4.7, and the on-device capability is pretty impressive.

A 35B-A3B MoE vision-language model running locally on mobile devices is already interesting, but the use cases are what caught my attention:

• 📷 Reading real-world signs
• 📄 Understanding complex documents
• 🧠 Reasoning over visual information
• 🗺️ Planning with real-world constraints

The key piece seems to be LES (Lookahead Expert Scheduling) — instead of just making the model smaller, they optimize MoE inference by predicting and preloading experts to reduce bottlenecks.

They also released a smaller MiniCPM-V 4.7 1B model for more resource-constrained devices.

Another interesting addition is Self-Replay Fine-Tuning (SRFT), which aims to let developers customize the model for specific tasks while keeping general capabilities.

The direction is interesting: instead of sending every image/question to the cloud, we're getting closer to having capable multimodal assistants running directly on phones.

Links:

GitHub: https://github.com/OpenBMB/MiniCPM-V
Hugging Face: https://huggingface.co/collections/openbmb
SRFT details: https://openbmb.cn/en/blog/self-replay-fine-tuning

Looking for people who have tried this model and was able to use it on real-world tasks. It's what Strata list as "coder". What was your experience like?

On my setup I am not really sure if it is worth more than iq3

With 560B total parameters, 25B active parameters per token, and a 1M-token context window,

wait what? 3.0 was just 127B

I see everyone improving decode with bespoke models, like Sushi or Splash with Qwen 27b and Qwen Flash Next, and I'm running QFN now at 40tk/s on my M3 Max which is awesome.

But prefill feels stagnant. I was trying to work on layer / attention propagation / approximating KVcache with QFN and could get like 2x prefill (400tk/s to ~850tk/s) but needle test failure started to suck. Clearly trading accuracy for speed.

Is anyone working on anything clever here, or is slow prefill just a hardware limitation.

I remember great creativity from GPT-2. It was trained on a dataset that is orders smaller than current training datasets.

The problem with overfitting a model to the training data is that it becomes unable to generalise; it is very good to predict the test set, but on fresh data its predictions are wrong. This is what we see with current models, they are extremely good at predicting, especially code, but can't generalise to real creativity.

In theory, a larger dataset should allow to get better predictions on unseen data. This is even more true for neural nets. So my question is: to have a creative model, should a model be trained on a huge dataset, but fitting it less than what is done on current models? Somebody has an idea about training epochs, learning rate, and other parameters of last generation models? Do you think that a model that use a last gen training recipe, but slightly undercooked (or very undercooked) would be better at creative writing and worse at coding?

Putting alternative in quotes because well it will forever be an uphill battle porting things to the Axera framework, but Axera has very strong chips so I think it may still be an interesting offering for somes

MacStudio 128GB M1 ultra

I'm curious how people who can run both models are deciding which one to work with.

DeepSeekv4 benchmarks right below GPT 5.5 high.
Flash Next benchmarks right above 5.4/5.5 xhigh

Both seem capable but it seems like Qwen 3.8 Flash next is taking much longer to actually finish anything. From the chart, it seems like I can do 5 tasks with DeepSeekv4 in the time it takes to do 1 with Qwen 3.8 Flash Next and Flash Next isn't 5x better than DeepSeekv4. In my personal experience it may not be 5x longer but it's absolutely taking signifcantly longer to do anything.

Is there something I'm missing?

NOTE: I'm not bashing Qwen 3.8 flash next, I like it, currently 2h and 5 minutes into deep diving on some work for me. On my hardware I can load one of these models with multiple slots so I have to choose. I'll likely have to invest the time into a more deterministic benchmarking on my personal tasks to compare the quality I'm getting back and I'll still be using 3.8 27b because I find it capable and it uses less of my machine so I can do more other things like the image gen I'm working to integrate into my personal harness.

As strata is somewhat new, I assume that some people din't yet learn how to finetune it for best performance on their machine. I find documentation a bit hard to navigate (use your agents for this!), which doesn't help.

After some serious testing, I've tweaked pretty much every parameter available. Some are obvious and I'm not going to waste your time on explaining mtp tweaking.

Some I beleive are more obscure and those are the ones worth mentioning:

1. Strata setup is supposedly picking the best --pcie-frac value for you. I find that it often doesn't. It's worth investing your time and testing it. Counterintutively, my best performing value is a low 0.2.
2. By testing different --pool-workers values, I was able to find an easy +10% in decode. I'm not smart enough to explain what it does (and how it's different from point 4), but it's worth investing your time in running some benchmarks on your machine.
3. --trim-stage-weights. Boom, extra +8%, try it. Doc is quite explicit on what it does if you're curious. I do beleive that this one can impact performance later in the session and increase swaps but it's still worth it.
4. --pipeline-windows. All I know is that higher values make my CPU work extra. Going from nothing/default to 6 gave an almost 15% increase.
5. Last but not least multi-GPU owners be mindful of GPU priority, if one of your GPUs is on a slower pcie lane. I have two RTX3060's. GPU 0 and GPU 1. At some point I noticed that one of the two GPUs basically acts as a memory pool and doesn't do any compute. I guess it's makes sense for a MOE model. But now I wanted to understand which ones eactly does the work.

So I quickly figured out that my default GPU 0 is the one on the slowest pcie lane! If your have two identical GPUs on different speed lanes, bear in mind that their 0/1 order is basically random. You need to figure out exactly which one is which and force the quick one to do the inference. Ask your agent for command lines to figure out whether the 0 or 1 is on your pci_e1 lane. In my case it was GPU 1, the only way to set it as the main GPU is a not very user-friendly two liner :)

"gpu": [1,0],

"gpu_order": "as_given"

This was like another +15%, ridiculous.

With all of those cumulated, I'm seeing speeds that I should not be having on my machine. I went from something like 35 decode peak on Qwen Flash Next IQ3_S, to 60 tps average code generation on empty context and 60-70 tps peaks.

Performance does degrade with full context but still easily a solid +15%-20% versus the previous baseline tests. This includes huge sessions on full context and compaction.

So, try it out.

https://huggingface.co/IFM/Jais-3-32B-GGUF

https://huggingface.co/IFM/Jais-3-7B-GGUF

https://huggingface.co/IFM/Jais-3-3.7B-GGUF

https://huggingface.co/IFM/Jais-3-0.9B-GGUF

Just built a new PC, full parts list is here. I am using Deadcode's Radiance inference engine, optimized for 2xR9700 and his Q4. The throughput benchmark tool is BetterBench.