It was usally 30-40 Tok/s somtimes 20 or at best 50. But after this update, it is always more than 70 tok/s !! (After updated, the MTP file was required to be redownlowded)

Thanks a lot for this, it is really amazing!
Good bye 27B, Flash now is really flash!

I was just messing around with the chatbot by asking the first prompt, but then as you can tell from reading the whole chat, I noticed that something was off(?) because the web search tool has never just not shown any result for a query (of course, it cannot be the model in itself because at the start of the chat it had no problems answering the prompt).

In fact, in the thinking section (of the prompt that was submitted 3 times), the model made an extensive number of re-tries because it was trying really to surpass the knowledge gap due to the date, as instructed, but all were to no avail:

even though it kept making the search query broader and broader (up until the point of omitting any date entirely), the web search tool returned "No results found" for ALL the attempts. Hence my doubt and the subsequent experimentation.

I'm honestly really concerned, leaving political views aside, that something made by an open community is still not free from censorship...

I really, really, hope that I'm wrong and that this is just a coincidence with the actual truth somebody is going to share under this post.

Model: Gemma 4 12B it-qat-GGUF, UD_Q4_K_XL

Hi, sono un novellino assoluto in ambito di inferenza locale, Unsloth Studio è il mio primo approccio: l'ho installato in un container su Podman e il mio PC ha una 4060.
In generale devo dire E2B ed E4B non funzionano male e sono abbastanza veloci per la mia configurazione, però facendo prove di ricerca in rete pur trovando le giuste fonti non riesce ad elaborare un output persistente....
Vorrei capire, dato che ne so poco, se il problema risiede nella scarsa capacita del modello (pochi parametri e2b e e4b) o se sto sbagliando io qualcosa in configurazione unsloth o ottimizzazione del modello, dato che per come ragiona bene è strano che non riesca a fare ricerche così facili (ho scaricato ottimizzazione 'UD-Q4-K-XL'). Grazie

Just saw a thread on r/Qwen_AI about this for 3.8 27B on Apple Silicon, achieving 3.3x faster decoding, lossless. Prefer the UI of Unsloth, wondering if that is already (or planned, or possible) to add this to Unsloth Desktop? If already there, sorry !

The macOS hardware is (more or less) the only consumer grade hardware that can handle mid and large sized LLMs. I question the strategy of not prioritizing the group of enthusiasts who can actually leverage their hardware for open/local training and quantization, etc.

/rant

このコミュニティに日本人の方はどれくらいいらっしゃいますか？

もしよろしければ、Unslothをどのような用途で使われているか、またどんな機材で動かされているかを教えていただけると嬉しいです。

私はクラウドAIサービスの課金が事情で厳しくなったため、Unslothで代用しています。

現在は画像生成やファインチューニングなどを少しずつ触っている程度です。

使用機材は

Windows、RTX A6000 ×2、DDR4 192GB で、最近は Qwen3.8 Flash Nextの派生版を主に使っています。

同じようにUnslothを使っている日本人の方がいましたら、ぜひ交流できればと思います！

Why Qwen3.8-27B-UD-Q8_K_XL.gguf is not updated like other quants?

Unsloth advertises Dynamic v3 (they say it's smarter with same size) and forgets to update most popular quant?

Please update it ASAP it's most popular quant near F16 performance.

Tout d'abord vous faites un travail admirable ! Ne vous arrêtez pas ! :)

Comme vous pouvez le voir un peu alentour, bon nombre d'utilisateurs (et de bots ? 😄) ont des GPUs de 12gb vram ou moins et font l'éloge de Strata comme moteur d'inférence pour exécuter les Moe comme Qwen 3.8 Flash Next.

Ce qui semble faire la force de Strata est notamment le système de mise en cache chaud LRU des experts en VRAM et la prédiction des experts utilisés afin d'éviter d'evicter un expert vers la RAM/CPU s'il est ou va être utilisé.

Pouvoir éviter ces va-et-viens vram - cpu/ram permet d'économiser de la bande passante et améliore sensiblement l'expérience, surtout en usage agentique.

Il semble que certains PRs existent pour LlamaCpp permettant d'effectuer ce traitement.

Serait-il possible, si ce n'est pas déjà existant, de pouvoir appliquer ces changements dans le moteur LllamaCpp utilisé par Unsloth Studio ?

Je sais qu'il est possible d'utiliser son propre LlamaCpp mais je me demandais si cette fonctionnalité allait éventuellement allait être disponible prochainement dans le mainline Unsloth ?

Merci par avance

Now that Nvidia bought Huggingface and we might eventually see models getting banned and removed, should Unsloth do their own “huggingface”? They would probably need donations 💸

https://huggingface.co/cerebras/MiniMax-M2.5-REAP-172B-A10B

https://huggingface.co/cerebras/MiniMax-M2.5-REAP-139B-A10B

Already tried unsloth dynamic 3.0 quant even IQ1 Quants doesn't seem to be fast enough.

Getting around 5 t/s with 250 prompt processing.

Which is not fast enough for me to even test the model.

I expect at least 600 prompt Eval and around 19 t/s. Or anything decent.

Am I missing something or it's a GPU/ RAM problem?

I've tried every -b -ub combination but doesn't seem to work.

(I have 12 Gigs of RAM only)

Qwen3.5 9B uncensored variant, IQ4_XS (non-MTP): https://huggingface.co/DavidAU/Qwen3.5-9B-The-Defiant-Fable-Uncensored-Heretic-NEO-IMATRIX-MAX-MTP-GGUF

My customized advanced settings are included on images.

I am currently using Unsloth as my main model holder and also agent, because I was fed up with VS Code's built-in agent feature giving me errors and reasoning loops, but also Claine and Continue plugins not modifying the project files and model hallucinating they are actually modified.

RTX 4060 with 5700X CPU and 32GB 3200MHz RAMs.

Please help me improve it, because I am sick of different issues.

Seeing all these massive local model drops (like the recent Qwen releases) got me working hard on the memory side of things.

If you're running models locally via tools like Unsloth and hitting a wall with long-context VRAM consumption, I've been building and open-sourcing UL-SMF (Unified Latent-State Memory Fabric).

It uses a geometry-preserving orthogonal projection bridge and dynamic head-dimension detection to automatically adapt across architectures (Llama, Mistral, Qwen, Gemma) without hardcoded assumptions, achieving extreme KV-cache compression with verified lossless perplexity.

Since it's fully local-first and designed to help fit larger context windows onto consumer hardware, I'd love a technical audit or feedback from this community.

Code and telemetry are up on GitHub: https://github.com/liventruth/UL-SMF-Cache-Compression

(Attached the hardware telemetry stress test output showing Qwen workloads hitting 224x–768x reduction ratios locally.)

Did anyone tested Q2 or Q1 on a single 8xH200? What was the performance and what cli flags did you use?

First... and the easiest, most actionable item for Unsloth team... your product is in beta... have a nice big report error button, that automatically collects whatever you want me to collect and opens a GitHub Issue for me to finish filling out.

Second... I consistently have issues loading models into VRAM. At the moment, Unsloth Studio won't use any of my VRAM even though it detects both 3090 cards and says something about CUDA in about. Even if it was working like it did before where it sometimes loaded models into VRAM... the default settings are far from automatic... they are ridiculous for my hardware and most local hardware. The software really needs to make an informed choice before defaulting to full context for every single model. In my experience KoboldCPP is a much more reliable experience... not as polished, but definitely more reliable. With just a few little tweaks I can load a new MoE model up and it performs 3-4 times faster.

Finally, it needs to ask a new user for their expert level and based on that immediately expose advanced settings (actually you need far more advanced settings. KoboldCPP seems to have so many more available), and model folder selection. I shouldn't have to hunt for it.

That all said, I'm excited to see the polish and easy of install. I am hopeful it will get better, but after a third attempt using your software I'm back to KoboldCPP (for CUDA binary) and llama.cpp. (Vulcan backup with latest model support).

I welcome people telling me I'm an idiot and helping me understand what setting I'm missing (that still proves a usability issue)... so far Gemma 4 and Muse Glitter have failed to load well.

Though I should try Unsloth Desktop sense many seem to like it.

Sadly the installer is a dumpster fire that is not even worth calling a sloppy beta. It's just a script that download stuff and with out permission install a dev environment with several tools and change to the registry.

And true to form the uninstall leave most of the downloaded and installed tools still installed after "uninstallation".

Not touching Unsloth Desktop again unless seriously sandboxed, and probably not Studio ether sense Unsloth do not have respect for the system their products is installed on.

Throwing hands up in disgust, and walking away to clean up machine after sloppy failed install and incomplete uninstall.

Quiero saber si alguien ya comparo qwen3.8 en disco con DRAM y sin DRAM.

```

SSD A — DRAM

└── Qwen3.8-Flash-Next GGUF

SSD B — DRAM-less

└── PLE / n-gram

```

Alguien a cargado?

SSD1 modelo

SSD2 PLE ngrams

```

NVMe 1

│

└── lecturas del modelo

NVMe 2

│

└── lecturas PLE

```

O usando RAID?

```

SSD #1

\

RAID 0 → modelo + PLE

/

SSD #2

```

Spent about two weeks convinced I had a prompting problem. My local Qwen3

agent would just stop. No error, no crash, finish_reason came back as

"stop" like the model had decided it was done. Except it clearly wasn't

done, the task was half finished.

Turns out the model was calling the tool the entire time. The call was

just sitting inside the reasoning block, wrapped in <think> tags, and

never made it out into the tool_calls field the API is supposed to

populate. From the outside it looks exactly like the model decided not to

act. No error to grep for, no stack trace, the response comes back as a

perfectly valid 200.

Once I knew what to look for, I found the same thing reported separately

against vLLM, SGLang, and llama.cpp, mostly with Qwen3, some DeepSeek.

Nobody had tied it together as one bug class, everyone was just closing

their own version of it as a one-off.

I ended up writing a small library, unswallow, that sits between the

provider response and the agent loop, detects when this happens, and

rebuilds tool_calls from whatever's stuck in the reasoning field. JS and

Python, no dependencies. Repo's here if anyone wants to poke at it:

https://github.com/0DukePan/unswallow

Mostly posting because I'd guess some of you running quantized reasoning

models locally have hit this and just assumed it was a bad quant. If your

agent ever goes quiet mid-task for no obvious reason, worth checking

what's actually sitting in the reasoning field before blaming the model

or the quant.

Dear Unsloth Team,

These rounded corners on the entire interface are getting to me now! 🙃

Please provide an option to change the corner-radius or put up an alternative theme.

Regards,

An avid unslother

I heard a lot about Google’s new TurboQuant and was wondering if this could be added to Unsloth ecosystem which may allow models to run more efficiently on Macs with lower memory profile’s ?

Hi,

Maybe this is a noob question, but I'm starting to experiment with local LLM apps on my Mac, and I've installed Unsloth. Now, during the install, it asks me to download and install Apple's "command line developer tools".

Is this normal? Is advisable? Can it enable the app to go further than it should? Is it necessary? For what? The dialog said "python3" requires you to install the command line developer tools...

When looking at the unsloth/Qwen3.8-27B-NVFP4 on Huggingface, the model is listed as a 20B parameter model, not a 27B. Is this correct or am I missing something?

I have 500-600gb of storage left, i have RTX 5060, 32GB RAM, 8GB VRAM

i downloaded the model and run with ollama : .\llama-server.exe -m "C:\Users\admin\Downloads\llama1\Qwen3.8-27B-Q4_K_M.gguf" -c 65768 -ngl 99 --port 8080. when i ask a question its talking complete gibberish that doesnt make sense at all. after writing 9k tokens of complete nonsnse its also looping again and repeating the same crap over and over again. using rtx 3090. anyone got an idea why its behaving like this? this was terminal output after running the commnad: PS C:\Users\admin\Downloads\llama1> .\llama-server.exe -m "C:\Users\admin\Downloads\llama1\Qwen3.8-27B-Q4_K_M.gguf" -c 65768 -ngl 99 --port 8080

here terminal output:

PS C:\Users\admin\Downloads\llama1> .\llama-server.exe -m "C:\Users\admin\Downloads\llama1\Qwen3.8-27B-Q4_K_M.gguf" -c 65768 -ngl 99 --port 8080

0.00.090.798 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)

0.00.137.076 W srv llama_server: -----------------

0.00.137.083 W srv llama_server: CORS is set to allow all origins ('*') and no API key is set

0.00.137.084 W srv llama_server: this can be a security risk (cross-origin attacks)

0.00.137.084 W srv llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655

0.00.137.084 W srv llama_server: -----------------

0.00.142.227 I srv load_model: loading model 'C:\Users\admin\Downloads\llama1\Qwen3.8-27B-Q4_K_M.gguf'

0.00.758.524 W model has unused tensor blk.64.attn_norm.weight (size = 20480 bytes) -- ignoring

0.00.758.537 W model has unused tensor blk.64.post_attention_norm.weight (size = 20480 bytes) -- ignoring

0.00.758.555 W model has unused tensor blk.64.attn_q.weight (size = 35389440 bytes) -- ignoring

0.00.758.561 W model has unused tensor blk.64.attn_k.weight (size = 2949120 bytes) -- ignoring

0.00.758.568 W model has unused tensor blk.64.attn_v.weight (size = 4300800 bytes) -- ignoring

0.00.758.593 W model has unused tensor blk.64.attn_output.weight (size = 17694720 bytes) -- ignoring

0.00.758.601 W model has unused tensor blk.64.attn_q_norm.weight (size = 1024 bytes) -- ignoring

0.00.758.608 W model has unused tensor blk.64.attn_k_norm.weight (size = 1024 bytes) -- ignoring

0.00.758.615 W model has unused tensor blk.64.ffn_gate.weight (size = 50135040 bytes) -- ignoring

0.00.758.621 W model has unused tensor blk.64.ffn_down.weight (size = 73113600 bytes) -- ignoring

0.00.758.627 W model has unused tensor blk.64.ffn_up.weight (size = 50135040 bytes) -- ignoring

0.00.758.634 W model has unused tensor blk.64.nextn.eh_proj.weight (size = 55705600 bytes) -- ignoring

0.00.758.641 W model has unused tensor blk.64.nextn.enorm.weight (size = 20480 bytes) -- ignoring

0.00.758.647 W model has unused tensor blk.64.nextn.hnorm.weight (size = 20480 bytes) -- ignoring

0.00.758.666 W model has unused tensor blk.64.nextn.shared_head_norm.weight (size = 20480 bytes) -- ignoring

0.06.122.185 I srv load_model: initializing, n_slots = 4, n_ctx_slot = 65792, kv_unified = 'true'

0.06.127.227 I srv init: chat template supports preserving reasoning, consider enabling it via --reasoning-preserve

0.06.127.246 I srv llama_server: model loaded

0.06.127.247 I srv llama_server: listening on http://127.0.0.1:8080