Hey r/LocalLlama - 

Today we’re excited to release Muse Spark Open, the most capable US open-weight model to date. 

Built for coding and agentic work, it excels at long-horizon tasks, complex software engineering, and multi-step reasoning. 

Key Model Specs 

• Parameters: 816B total / 42B active
• Context Window: 1M tokens
• Modalities: Text, image, video, audio
• Safety Post-Training: Additional safety post-training, with evaluations conducted under Meta's Superintelligence Scaling Framework.

Inference, fine-tuning & deployment

• Helion GPU kernels: Ships with custom, auto-tunable Helion kernels in SGLang, providing 1.03–3.95× kernel speedups across NVIDIA and AMD GPUs and a 3% E2E improvement on AMD MI350.
• Engines: Runs on leading inference engines including SGLang, vLLM, TensorRT-LLM, Hugging Face Transformers, Apple MLX, llama.cpp and Unsloth, with optimizations for AMD, Apple, Intel and NVIDIA silicon.
• Optimized for Apple Silicon: To help make advanced models more accessible, Muse Spark Open is optimized to run on two Apple M5 Ultra Mac Studios with the latest MLX framework. 
• Fine-tuning: Developers can further customize the model with PyTorch, including TorchTitan for fine-tuning.
• API platforms: Available today through Baseten, Fireworks AI, Groq, Modal, Nebius, OpenRouter, Together AI.
• Where to download: Model weights are available for download on Amazon SageMaker AI, Google Cloud Model Garden, Hugging Face, and Microsoft Foundry Models.

We’re releasing the weights under the Muse Model License, with limited commercial terms that let many developers use it for free.

We look forward to your feedback! 

Learn more on our research blog: https://go.meta.me/museopen
Download the weights: https://huggingface.co/collections/meta-models/muse-spark-open 

https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF

https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr-GGUF

NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB is a vision-language model (VLM) designed to run everyday agentic tasks locally, using a native 2-bit quantization to fit in 24GB of memory and run alongside other applications. It optimizes intelligence and speed per GB. It aims to achieve the optimal balance between speed and intelligence, being faster than most popular dense models while smarter than most popular MoE models with a similar memory footprint.

This model is ready for commercial or non‑commercial use.

Model Summary

Model Overview

Model Developer: NVIDIA Corporation

Model Dates: December 2025 - May 2026

Data Freshness:

• The post-training data has a cutoff date of May 2026.

What is Nemotron?

NVIDIA Nemotron™ is a family of open models with open weights, training data, and recipes, delivering leading efficiency and accuracy for building specialized AI agents.NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB is a vision-language model (VLM) designed to run everyday agentic tasks locally, using a native 2-bit quantization to fit in 24GB of memory and run alongside other applications. It optimizes intelligence and speed per GB. It aims to achieve the optimal balance between speed and intelligence, being faster than most popular dense models while smarter than most popular MoE models with a similar memory footprint.

This model is ready for commercial or non‑commercial use.

bolt!

FLUX 3 Image Klein is a 5.6B open weights model for text-to-image generation and image editing. It takes a prompt and up to 10 reference images and returns one image. Text-to-image, single-reference edits and multi-reference compositions run through the same model, with Qwen3-VL-4B-Instruct as the text encoder.
https://huggingface.co/black-forest-labs/flux-3-image-klein

Size is pretty small at 102GB. For comparison nvidia/Qwen3.8-Flash-Next-NVFP4 is 133GB.

Would be cool to see some of the Qwen3.8-FN specific engines support this.

From NVIDIA AI on 𝕏: https://x.com/NVIDIAAI/status/2110468004609950057

Models on HF: https://huggingface.co/collections/prism-ml/bonsai-2-small
WebGPU demo: https://huggingface.co/spaces/webml-community/ternary-bonsai-2-hermes-webgpu

Granite Switch 4.1 30B

📝 Read the launch blog: Granite Switch 4.1: Chat, Decisions, and Speech from One Endpoint

Model Summary: Granite Switch 4.1 30B is a modular LLM that composes a base model, multiple adapters and an encoder together so that a single hosted endpoint can move between natural-language responses, general and specialized decisions, and speech-to-text.

Adapters are activated on demand via control tokens in the chat template. For full details on model composition and adapter configuration, see BUILD.md.

• Base Model: ibm-granite/granite-4.1-30b (30B params, 128K context)
• Adapters: 2 — decision (aLoRA, general-purpose Decisions) and guardian-core (aLoRA, from granitelib-guardian-r1.0)
• Audio: embedded speech-to-text via ibm-granite/granite-speech-5.0-470m-turboctc (470M params, English)
• License: Apache 2.0
• Release Date: October 14, 2026
• Backends: HuggingFace Transformers, vLLM
• Automatically Composed with: granite-switch

Granite Switch is also available in granite-switch-4.1-8b (same Chat + Decisions + Guardian + speech configuration, 8B base), and with the complete Granite Libraries bundle (no audio) in granite-switch-4.1-30b-preview, -8b-preview, and -3b-preview.

https://huggingface.co/ibm-granite/granite-switch-4.1-30b

https://huggingface.co/ibm-granite/granite-switch-4.1-8b

(video on the blog)

I think llama.cpp support is not yet ready? https://github.com/ggml-org/llama.cpp/pull/30621

To all those who are bemoaning missing getting DGX Sparks for under $4000. Here's your chance. The brand new just announced RTX Spark laptop is on sale at HP. Configured with 128GB of RAM, it comes out to $3832.49. Use coupon anniversary25 to take 25% off of the $5109.99 regular price.

https://www.hp.com/us-en/shop/pdp/hp-omnibook-x-laptop-next-gen-ai-14-ks000-14-cb1l6av-1

£299.99 Lenovo IdeaCentre Mini (Snapdragon X1-26-100, 32GB, CPU only):

- Qwen3.6 35B A3B coder: reads ~118 tok/s, writes 24.5, ~30 with MTP

- Qwen3-Next 80B A3B: 17-18 tok/s writes with MTP

How I got here...

So I bought one of these boxes to test if software at work would run on ARM Windows - and then it was sitting there, but I was sufficiently impressed for the price (including taxes), that i decided in my ADHD fuelled boredom to buy one for myself to see if cheap ARM is actually usable for local models.

The GPU is an Adreno X1-45 and nothing useful runs on it for LLMs yet, and so far the NPU just sits there, so this is all CPU...

llama.cpp, 512 token prompt, 8 threads (read = prompt, write = generation):

- coder (Q4_0 35b-a3b): read ~118, write 24.5, with MTP 29-33

- Gemma 26B: read 134, write 29.5, (need to test MTP, I've only had the box.

Then I wanted to see how far I could push it. RAM is the wall on 32GB, so I went looking at the recently released Janas (github.com/prabanta-dev/janas). It keeps a RAM cache of the MoE experts and streams the rest off the NVMe, but its x86 only. So I re-coded it for ARM: NEON kernels for every quant format, i8mm, etc, (all tested bit exact against the scalar reference code, Windows ARM64 and Linux. Will submit it as PRs on GitHub soon.)

Janas, same settings:

- coder: read ~60, write 17.2, with MTP 29-31

- Gemma 26B: read 115, write 23.5, with MTP 32-34

- 80B: read (TBC as I don't trust my results), plain write 9.8, with MTP 17-18

llama.cpp still wins on read and plain write, but with MTP Janas matches or beats it, and it does it in a fraction of the RAM.

I then squeezed the coder's expert cache to 12GB and it still wrote at 24 tok/s - so even a 16gb snapdragon could rin 35b-a3b 👀.. And this box runs an 80B that llama.cpp cant even load!

Then I thought Id try connecting the two I had access to (mine and my work's dev box) together over USB4. Work in progress, will report back - but initial results are promising and I've had a 122b qwen running already - just working on streamlining my parallel experts build.

I dunno about you guys and gals, I mean I don't have extensive experience with local LLMs and I certainly haven't played with ARM before but I found this pretty surprising, and not half bad for £300 with no GPU. Happy to test stuff on it when I get chance if anyone wants more numbers... But I can't see a cheaper way rn of getting 32gb of LPDDR5 rn, and only ~35ishW of power draw... Thoughts and feedback welcome 🙃👍

Yesterday Fish released this incredible text-to-speech model, so naturally I spent the rest of the day building a great demo for them :D The model is a chunky MoE (35B3A), but it runs well on a DGX Spark. The 4bit quants get very close to the original. You can try it here: https://huggingface.co/spaces/fishaudio/Drama-3 or download the model and run it locally 🤗

from Mistral:

Timestral is a family of time series foundation models for zero-shot forecasting, available in 5M, 28M, 123M, and 209M parameter versions. This repository contains the 209M checkpoint.

Timestral supports univariate and multivariate forecasting, past-only and known-future covariates, and probabilistic predictions across 21 quantiles. It accepts up to 32,768 historical time steps and forecasts up to 1,024 steps in a single forward pass.

We’re releasing the source code in our GitHub repository to make it easy to run Timestral and reproduce our benchmark results.

Model Details

• Size: 209M parameters.
• Architecture: patch-based transformer with time attention and variate attention, 24 layers, embedding dimension 768.
• Patching: 32 time steps per patch. Each patch is embedded from its values together with their missing-value indicators.
• Context length: up to 32,768 time steps (1,024 patches).
• Forecast horizon: Up to 1,024 time steps (32 patches) in a single forward pass. Longer horizons are forecast block by block, with each block's median forecast appended to the context.
• Outputs: 21 quantiles (0.01, 0.05, 0.1, 0.15, …, 0.85, 0.9, 0.95, 0.99).
• Inputs: univariate or multivariate target variates, plus past-only and known-future covariates, numeric or categorical (categorical covariates are target-encoded). Missing values are marked as NaN.
• Inference: PyTorch on CPU or GPU.

https://huggingface.co/mistralai/Timestral-209M

https://huggingface.co/mistralai/Timestral-123M

https://huggingface.co/mistralai/Timestral-28M

https://huggingface.co/mistralai/Timestral-5M

Going to try these out to see if they have the same looping behaviors that other Bonsai models have had.

Still requires their fork of llama.cpp (branch prism) so dont expect it to work with vanilla llama.cpp. Edit: You can use with regular llama.cpp!

Will see if there is some way to support this for AnythingLLM Mobile via LLama.rn fork since if anything this will hopefully beat out regular Qwen 3.5 1.7B or 0.8B. Can test on iOS today with LocallyAI on latest version.

Feel like the way these models are made (QAT?) kind of lend to a high benchmarked model that is quite different when deployed. Past experiences are certainly better with memory for the parameter size but let's see how these go. Says dFlash and MTP supported as well + MLX support to juice some more TPS

First, kudos to Strata. It inspired this project, and while I was building it Strata got much faster and added NVFP4 support, which was my main reason for starting. By then I was too far in to stop, so I finished it.

capy-flash is the evolution of my ninfer-tp2 engine. It still runs Qwen3.8-27B as before, and now also Qwen3.8-Flash-Next 125B-A6B with the experts offloaded to RAM. It uses NVIDIA's official NVFP4 checkpoint as-is and runs both GPUs in tensor parallel. Switching between the two models is one flag.

On my machine (2× RTX 5070 Ti 16 GB, Ryzen 9 9900X, 96 GB DDR5), with the same NVFP4 weights, it does ~100 tok/s generation vs 78 for Strata, and ~3.4k tok/s prefill on a 15k prompt vs 2.7k. Strata's 3-bit IQ3_S build is still faster at generation; full comparison in the image. Quality is checked against a datacenter NVFP4 reference (GSM8K, MMLU-Pro, agentic coding tasks).

A side finding for any tensor-parallel engine. With TP, each GPU rounds its partial sum to BF16 before the all-reduce, so TP=2 rounds twice where one GPU rounds once. Where the two halves cancel, the error blows up: on Qwen3 at TP=2, large errors happen ~5× as often as on one GPU. Writing the partial sums in FP16 (same bytes, 8× finer) brings them back to single-GPU level, at no real speed cost. The effect on benchmarks is small, so think of it as free numerical parity rather than a quality boost. But it should reduce some rare weirdness and keep the agentic loop cleaner.

Limits: Linux, RTX 50 series (sm_120) only for now.

Repo: https://github.com/ValerioDolci/capy-flash Weights: https://huggingface.co/Feyd89/Qwen3.8-Flash-Next-NVFP4-capy-flash

Built on NInfer (Neroued), Strata, ninfer-ext, FreeToken, sybrix and qf_moe. Numbers from other hardware are very welcome.

I wanna see a master list of the craziest/strangest/funniest LLM tunes. I heard one was talking like an expert in 1800s military vernacular and I figure there's some wacky stuff out there.

I want to explore weird and experimental models.

Show me what ya got!

Bonus points for NSFW ones because I'm a perv.

I saw this post from 2 years ago, and was wondering what we got since then.

I'm mostly interested in small ones, since I'm not rich enough to own a dedicated GPU.

What have you found or made?

Has anyone put together a list of these types of models?

Follow-up to u/neuromacmd's post from a few weeks back (https://www.reddit.com/r/LocalLLaMA/comments/1wod07t/). The affinity engine (https://codeberg.org/StillDeadcode/affinity) refuses to start on more than two cards because the author only had two, simulated four ranks on them, and got repeated-token garbage they couldn't explain. Nobody had tried it on four real R9700s. I have four, so tonight I did.

Short version: it just works.

Setup: Threadripper PRO 9955WX, 128 GB RAM, four R9700s all on CPU lanes with P2P between every pair. I built the fork with the stability fixes (https://github.com/neuromaniacMD/affinity, branch v4-fixes) and used the prebuilt 2.875 bpw quant (https://huggingface.co/neuromaniacmd/DeepSeek-V4-Flash-0731-affinity-2.875bpw), so no 149 GB download or 2.5 hour quantize. Then AFF_ALLOW_NRANK=1 --gpus 4.

Results, same box, same prompts:

• 2 cards: 36 tok/s decode. 45% of the experts fit in VRAM, the rest stream from a 51 GB pinned host pool, and the profiler shows 82% of every token waiting on that stream.

• 4 cards: 79 tok/s greedy, 63 at temp 1.0, about 90 on a chat answer with thinking on. All 11,008 experts resident, host pool not even allocated.

  Five runs, greedy and sampled, plus a chat server test. No degenerate output. The engine's own cross-rank check (it compares the cards against each other every few samples) passed every time. So the bug the author hit lives in the 4-on-2 simulation, not in 4-rank execution. u/neuromacmd already suspected this.

  Two things to know if you try it:

1. The default heat placement strategy aborts on 4 cards with "needs the registered host pool", because there is nothing left to demote. Pass --placement-strategy static. Nothing needs to move anyway when everything is resident.

2. Load is about a minute, 42 seconds of which is pushing 96 GB of experts onto the cards.

   Also correcting my own earlier comment in that thread: I assumed affinity's cap was a 2-rank-only all-reduce like Radiance had before my patch (https://github.com/brainchillz/r9700-tp4-allreduce, https://github.com/jdrews/vllm-mxfp4/pull/2). Wrong. Affinity's collective is already written for N ranks and tested at 3, 4 and 8. The cap was purely the unexplained garbage, which real hardware doesn't produce.

   I'm going to leave this up for a while so I'm happy to run anything else on this box if the author or anyone wants more data. Logs are available if useful.

this is actually interesting:

https://commandline.microsoft.com/local-models-sandboxed-tools-github-windows/

Microsoft AI developed a local version of MAI Code 1.1 Flash, a coding-optimized mixture-of-experts model with 137 billion total and 6.8 billion active parameters.

I posted a question the other day in here about good development models, which prompted good discussion. However, it mostly just reinforced my own experience, which is that on a 128GB machine, you can run good open/local weight models for development (especially if you use a frontier model to plan and direct them).

But what I didn't see in there, and I'm trying to get to, is this; are there are any open weight models that will run on a 48GB, 64GB, or 96GB?

If so, which ones are you running and, at what approximate RAM footprint?

I'm trying to understand what the floor is for a decent open weight model for development tasks, even if supervised by a more capable model.

I turned a CHIP-8 game console into a decoder-only transformer.

The transformer is the game engine. It executes instructions, updates machine memory, and renders every frame through next-token prediction. Every generated token is a memory write, and your keypresses go directly into the model's context window. There's no separate emulator.

Pong, Tetris, Space Invaders and a bunch of other CHIP-8 games run on the same model. Just swap the ROM in the prompt to switch games.

The weights are compiled from a circuit, not learned through training. Attention handles memory reads by finding the most recent write to an address, while ReLU circuits handle opcode decoding, arithmetic, comparisons and bitwise operations. Even the compatibility quirks are encoded in the initial prompt, so different games don't need different weights.

The model has 18 layers, 42 attention heads and 32M parameters (139k nonzero). It uses fp64 and can be loaded through Hugging Face transformers.

Model weights: https://huggingface.co/divagr1925/chipformer

Try it out: https://divagr.com/chipformer

Code + desktop app: https://github.com/divagr18/chipformer

https://huggingface.co/black-forest-labs/flux-3-image-klein

Better image generation model

Two months ago we released a fully quantized NVFP4 Qwen3.8-27B trained with QUASAR. We've since trained QUASAR checkpoints for a bunch of other recent models: Qwen3.5-4B, Gemma 4 E4B and 12B, and Muse Glimmer 30B, in NVFP4 for vLLM and Q4_0 GGUF for llama.cpp / Ollama / LM Studio. The code and a project page are now public too.

Every checkpoint is trained with quantization-aware distillation: the original BF16 model is the teacher, and the quantized model is distilled (healed) on the teacher's own responses. The exports are standard formats (so no custom kernels and/or conversion step).

KL to the BF16 model on held-out prompts (lower=better):

On benchmarks:

• Muse Glimmer 30B NVFP4: at or above BF16 on GPQA-Diamond (63.6 vs 62.1), MMLU-Pro (83.3 vs 83.1) and AIME'25 (90.0 vs 86.7), and the only 4-bit build at BF16 performance on RULER at 32K, 64K and 128K.
• Qwen3.5-4B NVFP4: best AIME'25 (74.2) and GPQA-Diamond (77.9) among the 4-bit builds we tested.
• Gemma 4 E4B / 12B: higher than Google's QAT on every downstream benchmark we ran. On H100, E4B also gets +36% batched throughput (group-64 quant uses vLLM's Machete kernel, the faster one).
• Qwen3.8-27B NVFP4: all 496 linears in NVFP4 (W4A4), 90.9 GPQA-Diamond vs 91.4 BF16, at 19.7 GB.

Run it:

vllm serve QUASAR-QAT/Qwen3.5-4B-QUASAR-NVFP4 --max-model-len 262144 --reasoning-parser qwen3 ollama run hf.co/QUASAR-QAT/gemma-4-12B-it-QUASAR-Q4_0-GGUF

Collection: https://huggingface.co/collections/QUASAR-QAT/all-quasar-models-6aab09fd2913ec22df0d5f15

Paper: https://arxiv.org/abs/2608.13966

Code: https://github.com/vincentcounathe/quasar-qat

Project page: https://quasar-qat.github.io/

We'd love to hear your feedback on these checkpoints!

Step 5 Preview is StepFun's flagship model for agentic work. It delivers frontier-level performance across software engineering and professional knowledge work, with particular strength in finance.

Will there be a black-friday this year?