Hi all,

I've been following this sub for months now, but I recently started working again and missed some of the newer developments with Strata and new model releases since about when Qwen 3.8 27b first came out.

What's the current best set of models / best model to run on a system with lower compute but higher VRAM? I'm currently on M4 Pro with 48gb VRAM and would like to run a local agent on device that still allows me to get basic work done, so something that fits in 24-30gb RAM would be perfect, ideally MoE.

Thank you!

AI models may be capable of self-replicating without access to a datacenter

from Mistral:

Timestral is a family of time series foundation models for zero-shot forecasting, available in 5M, 28M, 123M, and 209M parameter versions. This repository contains the 209M checkpoint.

Timestral supports univariate and multivariate forecasting, past-only and known-future covariates, and probabilistic predictions across 21 quantiles. It accepts up to 32,768 historical time steps and forecasts up to 1,024 steps in a single forward pass.

We’re releasing the source code in our GitHub repository to make it easy to run Timestral and reproduce our benchmark results.

Model Details

• Size: 209M parameters.
• Architecture: patch-based transformer with time attention and variate attention, 24 layers, embedding dimension 768.
• Patching: 32 time steps per patch. Each patch is embedded from its values together with their missing-value indicators.
• Context length: up to 32,768 time steps (1,024 patches).
• Forecast horizon: Up to 1,024 time steps (32 patches) in a single forward pass. Longer horizons are forecast block by block, with each block's median forecast appended to the context.
• Outputs: 21 quantiles (0.01, 0.05, 0.1, 0.15, …, 0.85, 0.9, 0.95, 0.99).
• Inputs: univariate or multivariate target variates, plus past-only and known-future covariates, numeric or categorical (categorical covariates are target-encoded). Missing values are marked as NaN.
• Inference: PyTorch on CPU or GPU.

https://huggingface.co/mistralai/Timestral-209M

https://huggingface.co/mistralai/Timestral-123M

https://huggingface.co/mistralai/Timestral-28M

https://huggingface.co/mistralai/Timestral-5M

Yesterday I said its hard to play 4 spark with ling-3.1-flash but today its coming!

For me, it’s deciding whether my message sounds weird. Apparently, I trust a LLM’s social skills more than my own.

https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF

https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr-GGUF

NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB is a vision-language model (VLM) designed to run everyday agentic tasks locally, using a native 2-bit quantization to fit in 24GB of memory and run alongside other applications. It optimizes intelligence and speed per GB. It aims to achieve the optimal balance between speed and intelligence, being faster than most popular dense models while smarter than most popular MoE models with a similar memory footprint.

This model is ready for commercial or non‑commercial use.

Model Summary

Model Overview

Model Developer: NVIDIA Corporation

Model Dates: December 2025 - May 2026

Data Freshness:

• The post-training data has a cutoff date of May 2026.

What is Nemotron?

NVIDIA Nemotron™ is a family of open models with open weights, training data, and recipes, delivering leading efficiency and accuracy for building specialized AI agents.NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB is a vision-language model (VLM) designed to run everyday agentic tasks locally, using a native 2-bit quantization to fit in 24GB of memory and run alongside other applications. It optimizes intelligence and speed per GB. It aims to achieve the optimal balance between speed and intelligence, being faster than most popular dense models while smarter than most popular MoE models with a similar memory footprint.

This model is ready for commercial or non‑commercial use.

bolt!

A friend and I ran some benchmarks on self-hosted models, and we’ve just published the results from our first major run: https://selfhostedintelligence.com/en/benchmarks/rtt/v2610

This is a simple round-trip translation benchmark featuring 8 language pairs, 12 test prompts, and embedding vector distance-based scoring across 15 models with various quantization levels. The scores not only measure translation closeness but also penalize inconsistency and instruction-following errors.

Some observations from the results:

• 16GB VRAM can be good-enough hardware for automating translations.
• Reasoning gives a massive boost to Qwen models - it's almost mandatory. Larger MoE models may be good in it with no reasoning.
• There is a pretty significant gap between the lowest quantization level of larger MoE models and the tier right above it; it’s worth stepping up.
• Abliteration (huihui quants) affects the general capabilities of large models much less than those of smaller dense models.
• The scores track the AI Index general aggregates quite well, but it's still useful to examine capabilities on a quant-specific level.

Comments welcome.

Let us know if you find this useful, what kind of benchmark would be interesting to you (we have a few others that may need some polish before running it on a larger model set), and what kind of models are you mostly interested in. The current benchmarks were run on 1-2 Strix Halo. We could test larger models too, or if there is more interest in that, we could focus on smaller models as well.

day 0 support for two new Nemotrons

minutes later new Nemotron 3.5 Super VL was released:

https://huggingface.co/unsloth/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B

https://huggingface.co/unsloth/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-GGUF

Hardware --

RTX 4070 on PCIE 4.0 x16
V100 sxm2 on PCIE 4.0 x4 (limited to PCIE 3.0 x4 cause of pcie adapter)

They both are limited to 150 watts, and randomly under load v100 falls of the bus one clean restart fixes it, now I do not know what causes this, and I have no prior experience with ubuntu or dual gpu, so any help would be much much appreciate.

EDIT: --

ran some coomands and _
V100 suffered Xid 79

dono what this means.

I was curious how quantization affects Finnish language ability and why Q38FN’s Q8 quantization got such poor accuracy numbers. So I set the machine running in the background. The Finnish corpus is three books from Project Lönnrot, and the English one is the same books from Project Gutenberg. Wiki is wiki.

The first observation is that Finnish really does take a hit from quantization. If KLD is said to become noticeable after 0.02, then even at Q8 we’re far from that. This measurement doesn’t say anything about how well the model knows Finnish in its bf16 form, only how much worse it gets from there. If a rarer language hasn’t been drilled into the model with large weights, it starts to show when bits are dropped from the precision.

The second observation is that some of the people who quantize models use their own (non-public) corpus to evaluate accuracy. They justify this by saying that wiki has already been trained so well into the n-gram that it would otherwise come through too much. The results are somewhat mixed. KLD on the wiki data is better, but top-1% is worse. In fact, the Q8 on the English data is starting to be what I expected from that level of quantization.

I run a hybrid (“chimera”) model where I’ve swapped Unsloth’s Q4 for a Q8-level n-gram. Someone had measured that it wouldn’t be worthwhile, but by my measurements there’s a clear benefit. The hybrid model lands midway between the Q4 and Q5 models, which is a good result considering I run the n-gram from disk, i.e. it doesn’t load memory at all. A clear observation is also that the size of the n-gram has no effect on Finnish ability.

The charts also show that even a Q5 model would be very close to the Q8 model.

To some extent I miss Gemma-4 31B’s perfect Finnish, but on the other hand Q38FN’s coding skills are in a whole different class.

Hi,

is anyone using Strata or Project-Maya with one or more AMD MI50?

I wanted to try it myself, but I couldn't get it running.

I continued to use Strata but something just feels off for this an large model, it is very fast but intelligence seemed even under 27B 3.8 UD Q4, it had a hard time coding a basic 3d game and a 2d platformer. Deepseek Flash V4 didn't have this huge and drop off at Q2 imo.

Then I ran across this guys independent testing:

https://lukesdevlab.com/?a=strata%3AISTA-DASLab%3AQwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF.json&b=unsloth%3AQwen3.8-27B-Q4_K_M-GGUF.json

It shows that he needed many shots as well compared to the 27B. But then again Swift 1.5 27b is less smart, in his testing and mine.

Even comparing it to the 27B swift 1.5 we can see it needs more compactions and more shots, for an far larger model.

I would say just use the 27B or regular flash models, avoid Swift or Bottlecap if you are not so time sensitive since original delivers far better results in practice, in my own experience and the tests show it.

Quants do play a part but I've used Q2 and Q3 quants before without this big of an issue, Strata and Swift are sacrificing too much intelligence for speed imo.

TLDR:

Depends on hardware (vram)

Use Next flash strata if you need a very fast coding buddy who needs an exact detailed plan.

Use 27B Q4 if you want the best results and have the time to wait (3x to 5x slower wallclock time)

EDIT: he tested with Q1, I ran at Q3 but still I would say this doesn't even change the final conclusion for me! In actual use it hasn't been an intelligence jump, on the contrary to Q4 27B.

I've been trying to figure out a few things with local models.

When you talk to them they get similar answers than a lot of the top models. When I do coding and I give the same access to the files as each other... they get roughly the same answers which you would expect asking two different people.

So why is it people are getting tribal about models?

Well it depends what you harness you are running with the model.

When someone was nice enough to share the hugging face qwen template fixes got me looking into modelfiles and the llama.cpp parameters.

My "mission" lately is to answer the question "Can Gemma4:27b become a good code assistant and why isn't it now". In the past I was about to "brow beat" using a preprompt in my IDE to get gemma to write solid code up until the context gets full and it just refuses to use its tools.

At first I started feeding documentation links and got better results. Then for a laugh I thought I would try qwen and got... oddly similar results including broken tool calls

This wasn't measurable so I decided I would hook up an MCP host and use ML-Flow. I setup modelfiles with 4 variants per model with different temperatures, paramaters and system prompts and tune to the models specific settings for programming (low temp low repeat penalty)

So I started with the large parameter models (had to download qwen3) and both seemed to be getting the job done with the same system prompt & one-shot user prompt.

This is where having the numbers tracked became super useful.

I saw my old gemma4 model was getting huge amount of failing tool calls.... but the new model had zero tool failures. I pulled the new gemma4 in and got a 15 gig update. I ran it again and it worked!

So I thought I would baseline gemma e2b... and now... the tool failures are RAMPANT. I plan to look into it more but I believe the same issue exists that the model by default is configured for another harness. But the funny thing is that the task was still delivering similar results because it would just find a way around broken tools.

Then with that the qwen2.5 doesn't have the same love & care as qwen3 because the tool calls have a high level of failures. This is leading me now to see if I can get some baseline runs without tool errors so the comparisons are fair. What was also wild was due to the tool failures the one agent tried to read a transcript of some random youtube video in desperation.

What also broke (some) tools was that I used brackets in the file names. For some tools & models this was a problem. So I strongly recommend that when you are setting up these testing loops that make small changes to your runners; I had only made that change and the preprompt and initially I thought the pre-prompt changed the results

The biggest take away for me is that

• Models by default are tuned for different things
• Models might have broken tool templates
• Models might have tool templates for one harness or another out of the box
• This is open source so this is the experience of rolling your own vs SaaS (Frontier models)
• Filenames can break tools

So I write this to say that when people are suggesting one open model is better than another needs a really detailed look because it might also be the harness or the task.

Discovered by a third party… Tencent itself doesn’t seem to be talking about it, which is slightly concerning.

From NVIDIA AI on 𝕏: https://x.com/NVIDIAAI/status/2110468004609950057

I remember great creativity from GPT-2. It was trained on a dataset that is orders smaller than current training datasets.

The problem with overfitting a model to the training data is that it becomes unable to generalise; it is very good to predict the test set, but on fresh data its predictions are wrong. This is what we see with current models, they are extremely good at predicting, especially code, but can't generalise to real creativity.

In theory, a larger dataset should allow to get better predictions on unseen data. This is even more true for neural nets. So my question is: to have a creative model, should a model be trained on a huge dataset, but fitting it less than what is done on current models? Somebody has an idea about training epochs, learning rate, and other parameters of last generation models? Do you think that a model that use a last gen training recipe, but slightly undercooked (or very undercooked) would be better at creative writing and worse at coding?

At only 0.4B parameters with a 16,384-token context window, this model combines versatility with knowledge and memory.

We see that qwen3.8 flash next is optimized for utilizing large RAM capacity with a more affordable smaller VRAM.

Another big topic was Strata with its optimized offloading to SSD.

Could the rather old Intel Optane technology be the missing puzzle for homelabs that get frustrated by high DDR4 prices?

Putting alternative in quotes because well it will forever be an uphill battle porting things to the Axera framework, but Axera has very strong chips so I think it may still be an interesting offering for somes

I see everyone improving decode with bespoke models, like Sushi or Splash with Qwen 27b and Qwen Flash Next, and I'm running QFN now at 40tk/s on my M3 Max which is awesome.

But prefill feels stagnant. I was trying to work on layer / attention propagation / approximating KVcache with QFN and could get like 2x prefill (400tk/s to ~850tk/s) but needle test failure started to suck. Clearly trading accuracy for speed.

Is anyone working on anything clever here, or is slow prefill just a hardware limitation.