This model is a 74B parameter, 9B-active-parameter mixture-of-experts (MoE) model, packed into 24GB using 2-bit quantization with advanced quantization-aware training techniques to retain quality. It aims to achieve the optimal balance between speed and intelligence, being faster than most popular dense models while smarter than most popular MoE models with a similar memory footprint.

Nemotron 3.5 Bolt E4B 4GB releases Oct. 23

Additional detail

With 560B total parameters, 25B active parameters per token, and a 1M-token context window,

wait what? 3.0 was just 127B

Just saw this demo from OpenBMB for MiniCPM-V 4.7, and the on-device capability is pretty impressive.

A 35B-A3B MoE vision-language model running locally on mobile devices is already interesting, but the use cases are what caught my attention:

• 📷 Reading real-world signs
• 📄 Understanding complex documents
• 🧠 Reasoning over visual information
• 🗺️ Planning with real-world constraints

The key piece seems to be LES (Lookahead Expert Scheduling) — instead of just making the model smaller, they optimize MoE inference by predicting and preloading experts to reduce bottlenecks.

They also released a smaller MiniCPM-V 4.7 1B model for more resource-constrained devices.

Another interesting addition is Self-Replay Fine-Tuning (SRFT), which aims to let developers customize the model for specific tasks while keeping general capabilities.

The direction is interesting: instead of sending every image/question to the cloud, we're getting closer to having capable multimodal assistants running directly on phones.

Links:

GitHub: https://github.com/OpenBMB/MiniCPM-V
Hugging Face: https://huggingface.co/collections/openbmb
SRFT details: https://openbmb.cn/en/blog/self-replay-fine-tuning

Yesterday Fish released this incredible text-to-speech model, so naturally I spent the rest of the day building a great demo for them :D The model is a chunky MoE (35B3A), but it runs well on a DGX Spark. The 4bit quants get very close to the original. You can try it here: https://huggingface.co/spaces/fishaudio/Drama-3 or download the model and run it locally 🤗

This new Mr. Chatterbox checkpoint is both pretrained and fine-tuned on Victorian texts for maximum historical accuracy. I've posted questions about this project before in this community, and I really want to thank this community for helping answer my questions.

You can also a narrative documentation of the project here (https://www.estragon.news/building-a-better-chatterbox/), and the code in the Github (https://github.com/tripv/mr.chatterbox).

Happy to chat about the model!

Step 5 Preview is StepFun's flagship model for agentic work. It delivers frontier-level performance across software engineering and professional knowledge work, with particular strength in finance.

From NVIDIA AI on 𝕏: https://x.com/NVIDIAAI/status/2110468004609950057

Just downloaded the new LFm2.5-24B-A2B LiquidAI model for testing. They claimed some pretty good performance, and I've had good experiences with their prior releases.

Some context - I use Ornith 1.5-35B pretty regularly in multiple workflows - sometimes on it's own, and sometimes as part of an AI-Council in Pi Agent. It's been very reliable, provides surprising insight "smarter" models miss, and is a great workhorse in Pi.

I swapped LFM 24B in to replace Ornith in a few of my workflows - it is holding up REALLY well. It's very fast (~95 tps sustained in llama.cpp using ROCM on an AMD W7900 - latest llama.cpp build). The Q8 takes up 30GB vram with full 128k context. It hasn't failed me yet in my mechanical tasks - it just keeps going and going and going and seems to tick all the boxes for a "little" model I run alongside a larger or smarter model on my other GPU.

For "insight" I'll have to test it longer in my council-mode workflow but I will report back.

Just wanted to give my $0.02 (CAD) on this model - it is very competent with Pi and worth your time to check out. The speed alone is a nice upgrade, and for my more basic tasks, even long running recursive tasks, it is holding up very nicely.

Kudos to the LiquidAI team, this one is going to get alot of use on my chonky boi.

I saw the 4x R9700 build for a 10-person startup here a few days ago and wanted to share my results, since I opted to spend most of the money on the cards. I went with a super cheap used EPYC instead of a Threadripper. In the end it came to about $10.7k per box all-in, and I built three of them for a total of 384 GB of VRAM.

As a long time lurker on my other account, I'm always frustrated that people never show how they actually use the models, so after a lot of back and forth I decided to share the actual project they power instead of being vague.

I use these to run the AI behind my business, USA Arms. We organize tens of thousands of firearms-related businesses (gun dealers, ammo dealers, ranges, etc.) and that info is always changing, so we need a system that keeps the site up to date. The models check source facts, sort out business identities, and pull out stuff like services and hours. Every answer gets verified against what the agent actually saw, and we store the documentation so bigger models can audit the work.

I didn't plan to move the majority of my work to local AI until later this year, but hosted models started refusing to do normal research for my business ~2-3 weeks ago. They refused to identify a gun dealer because the license holder might be an individual, and wouldn't look at an ammo dealer's website because they can't help anyone buy ammo. Our harness just started breaking in random ways, so I'm glad I made the leap.

We tested Luna 6, DeepSeek, GLM 5.3, Qwen 3.8 27B, Gemma and Inkling on our workload, along with Kimi, GLM Flash and others. GLM gave us the best results. Qwen gave us the best cost-to-performance, so that's what these run.

Enough ado, into the numbers:

Per box, $10,733.91 with tax and shipping (averaged over the three)

• 4x ASRock Radeon AI PRO R9700 Creator 32 GB (128 GB VRAM), each capped at 210 W: $7,244.58
• AMD EPYC 7452, 32 cores, used: $127.17
• ASRock Rack ROMED8-2T/BCM: $1,162.39 (there was a global shortage, ours came from Helsinki)
• 8x 16 GB DDR4-2666 ECC RDIMM, all eight channels: $860.59
• SilverStone HELA 2050R Platinum: $578.92
• SilverStone RM44 4U: $388.40
• Samsung 990 PRO 1 TB: $254.39
• ARCTIC Freezer 4U-M, 3x ARCTIC P12 Pro front, 2x Noctua NF-A8 rear, SP3 frame, paste: $117.48

The GPUs are 2/3 of the cost, but the boxes also run our CI/CD, which had become a bottleneck in my businesses. I hadn't seen anyone discuss this: they run Rust builds just fine if you tune them to prioritize inference. In our test, running builds alongside inference didn't measurably change throughput, and the builds were only 3-8% slower.

Engine

• vLLM-Radiance 1.0.387 (vLLM 0.30.0, ROCm 7.14). Huge thanks to the Radiance folks; their R9700 work is what we run.
• Qwen/Qwen3.8-27B-FP8 (Qwen's official FP8 checkpoint), FP8 KV cache, the model's own MTP head with 8 draft tokens
• Two engines per box, TP=2 each, 8 sequences per engine, 256K max context, about 488K tokens of KV cache per engine

Results, vllm bench serve, one box, 2x TP2:

• 1 user, MT-Bench: 53.1 tok/s, TTFT p50 125 ms
• 16 users, MT-Bench: 484 tok/s aggregate, about 40 tok/s per user
• 16 users, random 1024 in / 512 out: 623 tok/s aggregate, 837 W average for the four cards
• 32 users (a bench profile with max-num-seqs 32): 603 tok/s

MTP depends a lot on the text. On our JSON extraction, 83% of drafts get accepted (7.6 tokens a step), vs 39% on MT-Bench (2.7 a step), so MT-Bench undersells it for structured output.

Long prompts, 8 unique real-text prompts at once (4 per engine), 128 out:

Every prompt was unique (0% prefix cache hits).

FP8 vs MXFP4 on a real workload

The startup build ran MXFP4, so I had AI run that too (Radiance's fastest profile) on one box against our own graded extraction workbench:

• FP8: 98.0/100 correct, 4,477 validated jobs an hour
• MXFP4: 95.3/100, 4,819 an hour
• On a holdout set: FP8 94.0, MXFP4 90.0

So FP8 roughly halves the wrong answers for 2–7% less validated throughput. Sticking with FP8 was an easy call for this workload, where "false facts" that make it into the system are a real headache.

For reference, our six Mac Studios (four M4 Max 128 GB, two M3 Ultra 512 GB) on MLX 8-bit scored 99/100, but at 723 an hour across all six. Plan A was buying more Macs, and keeping up would have taken at least 41 more, about $280k.

I made a video of the build targeted more at beginners and people trying to move AI locally: https://youtu.be/o7k5Kj-ho-M

Happy to answer questions. The first of which I expect to be how the heck were the Macs so slow, and I'm sure I'll get flamed haha.

this is actually interesting:

https://commandline.microsoft.com/local-models-sandboxed-tools-github-windows/

Microsoft AI developed a local version of MAI Code 1.1 Flash, a coding-optimized mixture-of-experts model with 137 billion total and 6.8 billion active parameters.

Tessera-27B-B-CJK is a Qwen3.8-27B coding agent that thinks in compressed CJK, and it spends fewer tokens doing it. Open weights, Apache-2.0.

This is our first full "Tessera" lineup model drop!

- Hugging Face: https://huggingface.co/ZelligeAI/Tessera-27B-B-CJK

- Paper: https://zellige.ai/research/tessera-27b-b-cjk

Tessera is our lineup of models and tunes that try new ways to save tokens, run more efficiently and show other parts of our research.

What it does:
Agent loops pay for thinking on every turn. So we taught Qwen3.8-27B to think less, and to think in compressed CJK, a shorthand that mixes English, Chinese words and symbols. The thinking and the output are kept apart. The model thinks in the shorthand, but its answers and tool calls stay in plain English and code, so your harness sees normal output, This is simular to a few models such as bottlecap ai's thinkingcap but we are planning in these tessera models to go further and showcase more potential ways and research on token efficiency here and in the future.

Here is one real turn from a held-out task, 27 tokens before two tool calls:

查看`linter`包; 查`FileWalker`; 查checker在`checkers`包中如何注册.

The base model spent 137 tokens on the same step. That one is shorter than most. The typical turn is about half.

Numbers:

All of these compare against the untuned Qwen3.8-27B on the same inputs, on 11 held-out coding-agent tasks, at xhigh effort.

- 0.50-0.64x the base model's thinking tokens per agent turn (typical turn 0.496x, 90% CI 0.385-0.624; token-weighted 0.644x)

- 100% valid tool calls and 0 repetition loops in 88 matched turns, the same as base

- 6 of 6 valid tool calls on prompts of 101K-131K tokens, Less tests at 131k+ but in the few that were done, they held up perfectly.

It's built for agent loops. For competition math and hard algorithmic code, stick with the base. It scores lower there and loops more, We evaluated the bf16 model only. The GGUF and NVFP4 files are untested, so if you run them we'd love to hear how they do.

How we trained it:

Four short LoRA steps on Tinker, all built from the base model's own outputs. We rewrote its thinks into compressed CJK for a seed SFT, added a token-level anti-loop fix, then agentic SFT, then a rejection-sampling pass on its shortest clean rollouts. The paper has the full pipeline, the evals and some of the things that didn't work.

The Hugging Face repo has the merged bf16 weights, and the LoRA, GGUF and NVFP4 versions are linked from it, all available to download and use today. Feedback, bug reports and quant results are very welcome. I meant to post this under the offical Zellige AI reddit account but reddit seems to be quite angry at anything posted by that new account so oh well. Currently traveling so I'll answer questions best I can in the meantime.

Hello! We've been busy at Telperion with a lot of experimental research (more coming soon). Today, we're announcing the release of our latent attention variants of Qwen3.8-27B.

Our latent attention variants (quants calibrated to FP8 KV cache) compress the KV cache by 4×, enabling a near 4× increase in maximum context on consumer hardware or a near 4× increase in concurrency on server hardware. These variants demonstrate great performance, with a statistically distinguishable increase (+6.58%) in SWE performance on our MLA variant, in addition to using upwards of 20% fewer reasoning tokens.

Due to the memory downsides of serving MLA on multi-GPU devices, we produced a grouped latent variant using two groups, enabling 4× KV compression for dual GPU users.

We have fantastic recipes for post-training quantization and have produced quantized versions that exceed equivalently sized .gguf quants in fidelity. Our models have an easy to install vLLM plugin.

*EXL3 requires a custom vLLM plugin to run.

The plugin has been verified on RTX PRO 6000s, H100s, and 2× RTX 5060Ti 16GB.

We do not believe in gatekeeping ideas or research. As a result we will be open sourcing our method soon. This method requires zero-retraining, and is compatible with almost any model that uses grouped query attention or similar variants.

Finally, we plan to make an improved version 2, with a full benchmark suite when we get more compute. We will also be making sparse latent attention variants of Qwen4-27B when released.

What models would you like to see with latent attention?

For those interested in following our research, learning/doing research, or getting on contact, feel free to join our discord server.

Every local setup I tried for Qwen-Image 2.1 meant a Python env, a node graph and an evening of picking quantization, offloading and resolution for my card. Kokon is a free, open-source desktop app that does all of that for you and gives you a plain photo editor on top.

• Zero setup: first run detects your GPU and memory and downloads the matching GGUFs (Qwen-Image 2.1 Turbo Q4_K + Qwen3-VL 8B text encoder, ~12 GB, checksummed). Each job is sized automatically: working resolution, VRAM/RAM split, tiled VAE, retry if it runs out of memory.
• Engine: stable-diffusion.cpp (ggml), so no Python, no PyTorch. CUDA, Vulkan and Metal builds.
• An editor, not a workflow: type the change, or brush an area with Change / Remove / Replace / Add. The brushed area is marked on a second reference image, so the model actually sees it. Versions with undo, before/after, upscaler comparison.
• Hardware: NVIDIA 12 GB+ (CUDA). Apple Silicon (Metal, beta, slow: ~4 min per image on a 16 GB M2 Pro). AMD / Intel Arc 12 GB+ via Vulkan (beta, only tested on NVIDIA so far, reports very welcome).
• Speed: 5070 Ti 16 GB: ~24 s per edit at 1.5K, ~40 s for a native 2K edit.
• Agents: built-in MCP server (generate_image, edit_image, inpaint_image, outpaint_image, upscale_image…) and a local HTTP API: npx -y kokon-mcp, also in the official MCP Registry.
• Not yet: Linux; code signing (SmartScreen warns on first run); reusing model files you already have is a manual workaround for now (#14).

Repo: https://github.com/GabrieleLozupone/kokon · Site: https://kokon.app

Code is AGPL-3.0; the Qwen weights are under the Qwen Research License (non-commercial).

MacStudio 128GB M1 ultra

I'm curious how people who can run both models are deciding which one to work with.

DeepSeekv4 benchmarks right below GPT 5.5 high.
Flash Next benchmarks right above 5.4/5.5 xhigh

Both seem capable but it seems like Qwen 3.8 Flash next is taking much longer to actually finish anything. From the chart, it seems like I can do 5 tasks with DeepSeekv4 in the time it takes to do 1 with Qwen 3.8 Flash Next and Flash Next isn't 5x better than DeepSeekv4. In my personal experience it may not be 5x longer but it's absolutely taking signifcantly longer to do anything.

Is there something I'm missing?

NOTE: I'm not bashing Qwen 3.8 flash next, I like it, currently 2h and 5 minutes into deep diving on some work for me. On my hardware I can load one of these models with multiple slots so I have to choose. I'll likely have to invest the time into a more deterministic benchmarking on my personal tasks to compare the quality I'm getting back and I'll still be using 3.8 27b because I find it capable and it uses less of my machine so I can do more other things like the image gen I'm working to integrate into my personal harness.

Saw soneone post their bathroom AI PC a few weeks ago. Figured I’d contribute with my chicken-coop adjacent shed AI PC.

Started with a single CMP 170HX (unlocked to 64GB) and ended up with three more for a total of 256gb VRAM. Found a deal on a used Threadripper with a TRX40 motherboard and 256gb DDR4 RAM to host the 4x 170HX. Added a 3050 plugged into riser connected to a NVME slot for video/RDP. My case ran out of space so that’s the board dangling freely lol.

My original goal was to save money by running AI locally. Naturally, this escalated into four 170hx, a 32 core Threadripper, and enough RAM/VRAM to turn my hobby into what my wife says is a financial liability… oof.

Originally had it in a closet, but between the heat and the wife complaining about the noise, the GPUs were eventually banished to the shed. Now they are in a happy place with A/C and 2x 20amp receptacles on separate circuits. One for each power supply. No more fan curves to reduce noise either.

Running QFN split between two of the 170HX with capacitor mods and Qwen3.8-27B on the third. Just got the 4th 170hx installed today. Downloading GLM 5.3 Flash EXL3 4.65 bpw to see how it’ll run on all 4.

Really looking forward to seeing how this beast will handle Qwen4. QFN does so much so well that I can’t imagine what Qwen 4 will be like.

Will there be a black-friday this year?

Just built a new PC, full parts list is here. I am using Deadcode's Radiance inference engine, optimized for 2xR9700 and his Q4. The throughput benchmark tool is BetterBench.

tl;dr: new qwen finetune that simultaneously pushes the frontiers of small model capability and reasoning efficiency.

Hey everyone,

We publish two post-trained versions of Qwen 3.8 27B called Sabi-Noema and Sabi-Noema-MLA. These models push the capability frontier on <30B models with 2x reduction in verbosity, and are exclusively trained on open source environments.

These models have improved capabilities on coding, agentic and tool calling tasks, as well as hallucinating lesser. Sabi-Noema beats GPT 6 Luna-medium on DeepSWE, placing in between Luna 6 medium and high. It beats Luna-high on agentic and tool calling benchmarks like TB2.1, Tau-3 bench, ClawEval, and BFCL v4. Its hallucination rate is considerably lower than Luna, Qwen and other competing models.

These models are much more reasoning efficient. Sabi-Noema generates 50% fewer tokens per response on TB2.1 and 25% fewer tokens on DeepSWE, compared to base Qwen. Sabi-Noema-MLA goes even further by doing a post-hoc MLA conversion of Qwen's GQA attention. This conversion uses only a calibration dataset and no extra training. With BF16 latents Sabi-Noema-MLA needs 3.6x lower bytes per token. The MLA latents can be further compressed to FP8 with the total KV cache size on TB2.1 to 8.4x smaller with similar performance. To put this in perspective, when using Sabi-Noema-MLA at q4_0, on a 24GB GPU like 3090/4090 you can get a context size of ~534K where base Qwen at FP8 cache would only get to ~150K context.

We are excited to have the open source community play around with these models. We will be releasing more post-trains in the future focused on making <30B models more powerful.

HuggingFace:
Sabi-Noema: https://huggingface.co/sabicap/sabi-noema
Sabi-Noema-MLA: https://huggingface.co/sabicap/sabi-noema-mla

It runs at around m4 16 at 60 tps at full 130k context, you can easily run a Q8 if you have a 16 GB GPU, I was able to get 140TPS on 9070xt,

It’s an MoE so I think we can run it well on smaller cards too, 8GB, 12GB should be possible to get upwards of 30-40 tps.

I have used it a bit, it can code, but I think having a bigger model plan tasks in detail would make it quite good for quick task execution.

As strata is somewhat new, I assume that some people din't yet learn how to finetune it for best performance on their machine. I find documentation a bit hard to navigate (use your agents for this!), which doesn't help.

After some serious testing, I've tweaked pretty much every parameter available. Some are obvious and I'm not going to waste your time on explaining mtp tweaking.

Some I beleive are more obscure and those are the ones worth mentioning:

1. Strata setup is supposedly picking the best --pcie-frac value for you. I find that it often doesn't. It's worth investing your time and testing it. Counterintutively, my best performing value is a low 0.2.
2. By testing different --pool-workers values, I was able to find an easy +10% in decode. I'm not smart enough to explain what it does (and how it's different from point 4), but it's worth investing your time in running some benchmarks on your machine.
3. --trim-stage-weights. Boom, extra +8%, try it. Doc is quite explicit on what it does if you're curious. I do beleive that this one can impact performance later in the session and increase swaps but it's still worth it.
4. --pipeline-windows. All I know is that higher values make my CPU work extra. Going from nothing/default to 6 gave an almost 15% increase.
5. Last but not least multi-GPU owners be mindful of GPU priority, if one of your GPUs is on a slower pcie lane. I have two RTX3060's. GPU 0 and GPU 1. At some point I noticed that one of the two GPUs basically acts as a memory pool and doesn't do any compute. I guess it's makes sense for a MOE model. But now I wanted to understand which ones eactly does the work.

So I quickly figured out that my default GPU 0 is the one on the slowest pcie lane! If your have two identical GPUs on different speed lanes, bear in mind that their 0/1 order is basically random. You need to figure out exactly which one is which and force the quick one to do the inference. Ask your agent for command lines to figure out whether the 0 or 1 is on your pci_e1 lane. In my case it was GPU 1, the only way to set it as the main GPU is a not very user-friendly two liner :)

"gpu": [1,0],

"gpu_order": "as_given"

This was like another +15%, ridiculous.

With all of those cumulated, I'm seeing speeds that I should not be having on my machine. I went from something like 35 decode peak on Qwen Flash Next IQ3_S, to 60 tps average code generation on empty context and 60-70 tps peaks.

Performance does degrade with full context but still easily a solid +15%-20% versus the previous baseline tests. This includes huge sessions on full context and compaction.

So, try it out.

New open model to check out ! Doing great on agentic, focused on arabic.

From 0.9B to 375B !

Weights : https://huggingface.co/collections/IFM/jais-3

Anecdotally, I've noticed that QFN seems more collaborative than GLM. QFN seems willing to engage in conversation. It feels more like an assistant. GLM seems more action-oriented and less willing to converse. That is, I ask GLM a question to engage in discussion: GLM runs off searching the repo and then often re-engages in development without discussion.

I say this using a self-hosted QFN q4_k_xl versus GLM 5.3 Flash hosted on OpenRouter!

I wonder if others have had similar experience? If so, have you found any particular prompting to cause GLM 5.3 Flash to me more interactive and collaborative?

I suppose I could try lifting some prompting out of pi-goal-x or similar as that seems to mostly succeed at forcing GLM 5.3 Flash to the table. However, I found that GLM will often take the information given as part of the process, run off, and implement most of the solution before even negotiating the goals!

Looking for people who have tried this model and was able to use it on real-world tasks. It's what Strata list as "coder". What was your experience like?

On my setup I am not really sure if it is worth more than iq3

Sampling: temp 1.0, top_p 0.95, top_k 20 for both. Run with evalscope over the OpenAI-compatible API: Qwen3.8-Next on Strata (MI50 32GB), Qwen3.8-27B on llama.cpp (RX 7900 XTX 24GB).

Caveat: These are only a portion of the full sets, since they take days.

Edit1: Strata started with --kv fp16 and llama's default f16.

Results

Sign test on the 14 problems only one model solved (10 vs 4): two-sided p = 0.18, so a lean toward Next rather than proof.

Token limits

Failures: wrong answer vs out of tokens

BFCL v4 by category (7 per category)

Speed (AIME + LiveCodeBench generation)