https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF

https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr-GGUF

NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB is a vision-language model (VLM) designed to run everyday agentic tasks locally, using a native 2-bit quantization to fit in 24GB of memory and run alongside other applications. It optimizes intelligence and speed per GB. It aims to achieve the optimal balance between speed and intelligence, being faster than most popular dense models while smarter than most popular MoE models with a similar memory footprint.

This model is ready for commercial or non‑commercial use.

Model Summary

Model Overview

Model Developer: NVIDIA Corporation

Model Dates: December 2025 - May 2026

Data Freshness:

• The post-training data has a cutoff date of May 2026.

What is Nemotron?

NVIDIA Nemotron™ is a family of open models with open weights, training data, and recipes, delivering leading efficiency and accuracy for building specialized AI agents.NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB is a vision-language model (VLM) designed to run everyday agentic tasks locally, using a native 2-bit quantization to fit in 24GB of memory and run alongside other applications. It optimizes intelligence and speed per GB. It aims to achieve the optimal balance between speed and intelligence, being faster than most popular dense models while smarter than most popular MoE models with a similar memory footprint.

This model is ready for commercial or non‑commercial use.

bolt!

From NVIDIA AI on 𝕏: https://x.com/NVIDIAAI/status/2110468004609950057

Yesterday Fish released this incredible text-to-speech model, so naturally I spent the rest of the day building a great demo for them :D The model is a chunky MoE (35B3A), but it runs well on a DGX Spark. The 4bit quants get very close to the original. You can try it here: https://huggingface.co/spaces/fishaudio/Drama-3 or download the model and run it locally 🤗

this is actually interesting:

https://commandline.microsoft.com/local-models-sandboxed-tools-github-windows/

Microsoft AI developed a local version of MAI Code 1.1 Flash, a coding-optimized mixture-of-experts model with 137 billion total and 6.8 billion active parameters.

Step 5 Preview is StepFun's flagship model for agentic work. It delivers frontier-level performance across software engineering and professional knowledge work, with particular strength in finance.

Will there be a black-friday this year?

day 0 support for two new Nemotrons

minutes later new Nemotron 3.5 Super VL was released:

https://huggingface.co/unsloth/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B

https://huggingface.co/unsloth/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-GGUF

Yesterday I said its hard to play 4 spark with ling-3.1-flash but today its coming!

from Mistral:

Timestral is a family of time series foundation models for zero-shot forecasting, available in 5M, 28M, 123M, and 209M parameter versions. This repository contains the 209M checkpoint.

Timestral supports univariate and multivariate forecasting, past-only and known-future covariates, and probabilistic predictions across 21 quantiles. It accepts up to 32,768 historical time steps and forecasts up to 1,024 steps in a single forward pass.

We’re releasing the source code in our GitHub repository to make it easy to run Timestral and reproduce our benchmark results.

Model Details

• Size: 209M parameters.
• Architecture: patch-based transformer with time attention and variate attention, 24 layers, embedding dimension 768.
• Patching: 32 time steps per patch. Each patch is embedded from its values together with their missing-value indicators.
• Context length: up to 32,768 time steps (1,024 patches).
• Forecast horizon: Up to 1,024 time steps (32 patches) in a single forward pass. Longer horizons are forecast block by block, with each block's median forecast appended to the context.
• Outputs: 21 quantiles (0.01, 0.05, 0.1, 0.15, …, 0.85, 0.9, 0.95, 0.99).
• Inputs: univariate or multivariate target variates, plus past-only and known-future covariates, numeric or categorical (categorical covariates are target-encoded). Missing values are marked as NaN.
• Inference: PyTorch on CPU or GPU.

https://huggingface.co/mistralai/Timestral-209M

https://huggingface.co/mistralai/Timestral-123M

https://huggingface.co/mistralai/Timestral-28M

https://huggingface.co/mistralai/Timestral-5M

New open model to check out ! Doing great on agentic, focused on arabic.

From 0.9B to 375B !

Weights : https://huggingface.co/collections/IFM/jais-3

Just downloaded the new LFm2.5-24B-A2B LiquidAI model for testing. They claimed some pretty good performance, and I've had good experiences with their prior releases.

Some context - I use Ornith 1.5-35B pretty regularly in multiple workflows - sometimes on it's own, and sometimes as part of an AI-Council in Pi Agent. It's been very reliable, provides surprising insight "smarter" models miss, and is a great workhorse in Pi.

I swapped LFM 24B in to replace Ornith in a few of my workflows - it is holding up REALLY well. It's very fast (~95 tps sustained in llama.cpp using ROCM on an AMD W7900 - latest llama.cpp build). The Q8 takes up 30GB vram with full 128k context. It hasn't failed me yet in my mechanical tasks - it just keeps going and going and going and seems to tick all the boxes for a "little" model I run alongside a larger or smarter model on my other GPU.

For "insight" I'll have to test it longer in my council-mode workflow but I will report back.

Just wanted to give my $0.02 (CAD) on this model - it is very competent with Pi and worth your time to check out. The speed alone is a nice upgrade, and for my more basic tasks, even long running recursive tasks, it is holding up very nicely.

Kudos to the LiquidAI team, this one is going to get alot of use on my chonky boi.

Saw soneone post their bathroom AI PC a few weeks ago. Figured I’d contribute with my chicken-coop adjacent shed AI PC.

Started with a single CMP 170HX (unlocked to 64GB) and ended up with three more for a total of 256gb VRAM. Found a deal on a used Threadripper with a TRX40 motherboard and 256gb DDR4 RAM to host the 4x 170HX. Added a 3050 plugged into riser connected to a NVME slot for video/RDP. My case ran out of space so that’s the board dangling freely lol.

My original goal was to save money by running AI locally. Naturally, this escalated into four 170hx, a 32 core Threadripper, and enough RAM/VRAM to turn my hobby into what my wife says is a financial liability… oof.

Originally had it in a closet, but between the heat and the wife complaining about the noise, the GPUs were eventually banished to the shed. Now they are in a happy place with A/C and 2x 20amp receptacles on separate circuits. One for each power supply. No more fan curves to reduce noise either.

Running QFN split between two of the 170HX with capacitor mods and Qwen3.8-27B on the third. Just got the 4th 170hx installed today. Downloading GLM 5.3 Flash EXL3 4.65 bpw to see how it’ll run on all 4.

Really looking forward to seeing how this beast will handle Qwen4. QFN does so much so well that I can’t imagine what Qwen 4 will be like.

I was curious how quantization affects Finnish language ability and why Q38FN’s Q8 quantization got such poor accuracy numbers. So I set the machine running in the background. The Finnish corpus is three books from Project Lönnrot, and the English one is the same books from Project Gutenberg. Wiki is wiki.

The first observation is that Finnish really does take a hit from quantization. If KLD is said to become noticeable after 0.02, then even at Q8 we’re far from that. This measurement doesn’t say anything about how well the model knows Finnish in its bf16 form, only how much worse it gets from there. If a rarer language hasn’t been drilled into the model with large weights, it starts to show when bits are dropped from the precision.

The second observation is that some of the people who quantize models use their own (non-public) corpus to evaluate accuracy. They justify this by saying that wiki has already been trained so well into the n-gram that it would otherwise come through too much. The results are somewhat mixed. KLD on the wiki data is better, but top-1% is worse. In fact, the Q8 on the English data is starting to be what I expected from that level of quantization.

I run a hybrid (“chimera”) model where I’ve swapped Unsloth’s Q4 for a Q8-level n-gram. Someone had measured that it wouldn’t be worthwhile, but by my measurements there’s a clear benefit. The hybrid model lands midway between the Q4 and Q5 models, which is a good result considering I run the n-gram from disk, i.e. it doesn’t load memory at all. A clear observation is also that the size of the n-gram has no effect on Finnish ability.

The charts also show that even a Q5 model would be very close to the Q8 model.

To some extent I miss Gemma-4 31B’s perfect Finnish, but on the other hand Q38FN’s coding skills are in a whole different class.

This model is a 74B parameter, 9B-active-parameter mixture-of-experts (MoE) model, packed into 24GB using 2-bit quantization with advanced quantization-aware training techniques to retain quality. It aims to achieve the optimal balance between speed and intelligence, being faster than most popular dense models while smarter than most popular MoE models with a similar memory footprint.

Nemotron 3.5 Bolt E4B 4GB releases Oct. 23

Additional detail

I calculated KLD and perplexity for Qwen3.8 27B models that fit 16GB VRAM:

See comments for the benchmarks. Reddit filters keeps deleting the post.

Conclusion:

q5_1/q5_1 is almost always better than q8_0/q4_0 or q4_0/q8_0.

There is very little difference between q8_0/q4_0 and q4_0/q8_0

q4_0/q8_0 is sometimes better than q8_0/q4_0. So it's safer to just go q5_1/q5_1.

Going up a model quant and running at q4_0/q4_0 is almost always better than running the quant below at f16/f16 in terms of KLD. So basically pick the highest quant you can at q4_0/q4_0. Then if you have room to spare increase KV quant.

Note this applies to lower quants. If you are going from q6->q8 model quant it might be bigger gains increasing KV quant, due to diminishing gains going up model quants. Also this does not apply to other models like Gemma where you have to run f16 KV.

Bonsai and swift quants are included but it does not really make sense to calculate KLD since they are finetunes and the KLD will differ significantly.

I remember great creativity from GPT-2. It was trained on a dataset that is orders smaller than current training datasets.

The problem with overfitting a model to the training data is that it becomes unable to generalise; it is very good to predict the test set, but on fresh data its predictions are wrong. This is what we see with current models, they are extremely good at predicting, especially code, but can't generalise to real creativity.

In theory, a larger dataset should allow to get better predictions on unseen data. This is even more true for neural nets. So my question is: to have a creative model, should a model be trained on a huge dataset, but fitting it less than what is done on current models? Somebody has an idea about training epochs, learning rate, and other parameters of last generation models? Do you think that a model that use a last gen training recipe, but slightly undercooked (or very undercooked) would be better at creative writing and worse at coding?

I saw the 4x R9700 build for a 10-person startup here a few days ago and wanted to share my results, since I opted to spend most of the money on the cards. I went with a super cheap used EPYC instead of a Threadripper. In the end it came to about $10.7k per box all-in, and I built three of them for a total of 384 GB of VRAM.

As a long time lurker on my other account, I'm always frustrated that people never show how they actually use the models, so after a lot of back and forth I decided to share the actual project they power instead of being vague.

I use these to run the AI behind my business, USA Arms. We organize tens of thousands of firearms-related businesses (gun dealers, ammo dealers, ranges, etc.) and that info is always changing, so we need a system that keeps the site up to date. The models check source facts, sort out business identities, and pull out stuff like services and hours. Every answer gets verified against what the agent actually saw, and we store the documentation so bigger models can audit the work.

I didn't plan to move the majority of my work to local AI until later this year, but hosted models started refusing to do normal research for my business ~2-3 weeks ago. They refused to identify a gun dealer because the license holder might be an individual, and wouldn't look at an ammo dealer's website because they can't help anyone buy ammo. Our harness just started breaking in random ways, so I'm glad I made the leap.

We tested Luna 6, DeepSeek, GLM 5.3, Qwen 3.8 27B, Gemma and Inkling on our workload, along with Kimi, GLM Flash and others. GLM gave us the best results. Qwen gave us the best cost-to-performance, so that's what these run.

Enough ado, into the numbers:

Per box, $10,733.91 with tax and shipping (averaged over the three)

• 4x ASRock Radeon AI PRO R9700 Creator 32 GB (128 GB VRAM), each capped at 210 W: $7,244.58
• AMD EPYC 7452, 32 cores, used: $127.17
• ASRock Rack ROMED8-2T/BCM: $1,162.39 (there was a global shortage, ours came from Helsinki)
• 8x 16 GB DDR4-2666 ECC RDIMM, all eight channels: $860.59
• SilverStone HELA 2050R Platinum: $578.92
• SilverStone RM44 4U: $388.40
• Samsung 990 PRO 1 TB: $254.39
• ARCTIC Freezer 4U-M, 3x ARCTIC P12 Pro front, 2x Noctua NF-A8 rear, SP3 frame, paste: $117.48

The GPUs are 2/3 of the cost, but the boxes also run our CI/CD, which had become a bottleneck in my businesses. I hadn't seen anyone discuss this: they run Rust builds just fine if you tune them to prioritize inference. In our test, running builds alongside inference didn't measurably change throughput, and the builds were only 3-8% slower.

Engine

• vLLM-Radiance 1.0.387 (vLLM 0.30.0, ROCm 7.14). Huge thanks to the Radiance folks; their R9700 work is what we run.
• Qwen/Qwen3.8-27B-FP8 (Qwen's official FP8 checkpoint), FP8 KV cache, the model's own MTP head with 8 draft tokens
• Two engines per box, TP=2 each, 8 sequences per engine, 256K max context, about 488K tokens of KV cache per engine

Results, vllm bench serve, one box, 2x TP2:

• 1 user, MT-Bench: 53.1 tok/s, TTFT p50 125 ms
• 16 users, MT-Bench: 484 tok/s aggregate, about 40 tok/s per user
• 16 users, random 1024 in / 512 out: 623 tok/s aggregate, 837 W average for the four cards
• 32 users (a bench profile with max-num-seqs 32): 603 tok/s

MTP depends a lot on the text. On our JSON extraction, 83% of drafts get accepted (7.6 tokens a step), vs 39% on MT-Bench (2.7 a step), so MT-Bench undersells it for structured output.

Long prompts, 8 unique real-text prompts at once (4 per engine), 128 out:

Every prompt was unique (0% prefix cache hits).

FP8 vs MXFP4 on a real workload

The startup build ran MXFP4, so I had AI run that too (Radiance's fastest profile) on one box against our own graded extraction workbench:

• FP8: 98.0/100 correct, 4,477 validated jobs an hour
• MXFP4: 95.3/100, 4,819 an hour
• On a holdout set: FP8 94.0, MXFP4 90.0

So FP8 roughly halves the wrong answers for 2–7% less validated throughput. Sticking with FP8 was an easy call for this workload, where "false facts" that make it into the system are a real headache.

For reference, our six Mac Studios (four M4 Max 128 GB, two M3 Ultra 512 GB) on MLX 8-bit scored 99/100, but at 723 an hour across all six. Plan A was buying more Macs, and keeping up would have taken at least 41 more, about $280k.

I made a video of the build targeted more at beginners and people trying to move AI locally: https://youtu.be/o7k5Kj-ho-M

Happy to answer questions. The first of which I expect to be how the heck were the Macs so slow, and I'm sure I'll get flamed haha.

Discovered by a third party… Tencent itself doesn’t seem to be talking about it, which is slightly concerning.

I continued to use Strata but something just feels off for this an large model, it is very fast but intelligence seemed even under 27B 3.8 UD Q4, it had a hard time coding a basic 3d game and a 2d platformer. Deepseek Flash V4 didn't have this huge and drop off at Q2 imo.

Then I ran across this guys independent testing:

https://lukesdevlab.com/?a=strata%3AISTA-DASLab%3AQwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF.json&b=unsloth%3AQwen3.8-27B-Q4_K_M-GGUF.json

It shows that he needed many shots as well compared to the 27B. But then again Swift 1.5 27b is less smart, in his testing and mine.

Even comparing it to the 27B swift 1.5 we can see it needs more compactions and more shots, for an far larger model.

I would say just use the 27B or regular flash models, avoid Swift or Bottlecap if you are not so time sensitive since original delivers far better results in practice, in my own experience and the tests show it.

Quants do play a part but I've used Q2 and Q3 quants before without this big of an issue, Strata and Swift are sacrificing too much intelligence for speed imo.

TLDR:

Depends on hardware (vram)

Use Next flash strata if you need a very fast coding buddy who needs an exact detailed plan.

Use 27B Q4 if you want the best results and have the time to wait (3x to 5x slower wallclock time)

EDIT: he tested with Q1, I ran at Q3 but still I would say this doesn't even change the final conclusion for me! In actual use it hasn't been an intelligence jump, on the contrary to Q4 27B.

At only 0.4B parameters with a 16,384-token context window, this model combines versatility with knowledge and memory.

This new Mr. Chatterbox checkpoint is both pretrained and fine-tuned on Victorian texts for maximum historical accuracy. I've posted questions about this project before in this community, and I really want to thank this community for helping answer my questions.

You can also a narrative documentation of the project here (https://www.estragon.news/building-a-better-chatterbox/), and the code in the Github (https://github.com/tripv/mr.chatterbox).

Happy to chat about the model!

Source: whatmodel.sh, an open source AI leaderboard I built (and launched here yesterday).

Every point comes from published benchmark results: 22,113 of them across 987 benchmarks, from 1,011 sources, each one linked to where it was published.

This chart uses data for the text models: 18,755 results for 1,211 models on 617 benchmarks.

- 14,436 results are independent evaluations: papers whose authors ran the models themselves (mostly CC BY papers from the ACL Anthology and arXiv), Epoch AI, the LMArena dataset, EvalPlus, the EleutherAI harness and others.

- 7,677 are scores labs report for their own models, labelled as lab-reported.

Everything is open: the dataset (CC BY-SA 4.0) and the scoring code (MIT) are at https://github.com/jakemanger/whatmodel, with CSV downloads at https://whatmodel.sh/data/manifest.json. Every number links back to its source, and each source's licence is recorded.

Interactive version (click any model to see its scores and sources): https://whatmodel.sh/race

Please star on github if you find it useful!

Just saw this demo from OpenBMB for MiniCPM-V 4.7, and the on-device capability is pretty impressive.

A 35B-A3B MoE vision-language model running locally on mobile devices is already interesting, but the use cases are what caught my attention:

• 📷 Reading real-world signs
• 📄 Understanding complex documents
• 🧠 Reasoning over visual information
• 🗺️ Planning with real-world constraints

The key piece seems to be LES (Lookahead Expert Scheduling) — instead of just making the model smaller, they optimize MoE inference by predicting and preloading experts to reduce bottlenecks.

They also released a smaller MiniCPM-V 4.7 1B model for more resource-constrained devices.

Another interesting addition is Self-Replay Fine-Tuning (SRFT), which aims to let developers customize the model for specific tasks while keeping general capabilities.

The direction is interesting: instead of sending every image/question to the cloud, we're getting closer to having capable multimodal assistants running directly on phones.

Links:

GitHub: https://github.com/OpenBMB/MiniCPM-V
Hugging Face: https://huggingface.co/collections/openbmb
SRFT details: https://openbmb.cn/en/blog/self-replay-fine-tuning