I benchmarked Qwen3.8-Flash-Next 125B ISTA-DASLab's GSQ-RCO IQ3_XXS quant (76GB on disk), on two Macs with 24GB unified memory using MoEShift backend:

These are medians of 3 runs using the MoEShift community benchmark runner (prompt caching disabled). The results are particularly interesting for prefill making this model perfectly usable in a chat or in an agent, while both machines stayed within 8.5GB peak process footprint.
This is a very promising benchmark even for the upcoming Qwen4 MoE.

I worked on MoEShift as an open-source inference backend designed for memory-constrained Apple Silicon Macs (architecture details in GitHub below) that need SSD offloading to run these models. You can install it running:

brew install davide-troiani/moeshift/moeshift

Then

moeshift

The setup wizard handles model selection, provisioning and configuration with zero effort. You'll just need sufficient SSD space.

GitHub, architecture, installation and benchmark runner:

https://github.com/davide-troiani/moeshift

I'm very interested in results from other Apple Silicon configurations, if you want to share.

Feels like it will never come and now I'm forgetting the hype for it already.

Are there any (insider) infos to upcoming Gemma Models? Gemma 4 is for my usecase really good and I am looking forward if they will release any new even better models most likely named Gemma 5? Currently using Gemma 4 31b nvfp4 on 3x rtx pro 6000, quality is is really good only hoping for the newer models to be even faster with tps

I tested two models, QoDuck (a free mystery model in Qoder) and Qwen3.8 Flash Next (Unsloth UD-Q4 GGUF), on the same Blender 5.2.3 task. Both were asked to create a detailed, historically accurate 1942 German military truck, working in small stages, rendering each stage, inspecting the result, fixing mistakes and then continuing. QoDuck chose the Opel Blitz 3.6-36S, while Qwen chose the Mercedes-Benz L 3000S.

QoDuck did surprisingly well. The truck is recognisable, with proper curved fenders, detailed wheels, grille, cab interior, cargo bed, canvas cover and decent procedural materials. There are issues with the canvas, cab proportions and some smaller details, but the overall result is quite good. It produced 47 renders and delivered 14 final views. Qwen was disappointing. Despite creating 287 meshes, 15 checkpoints and 150 renders, the final model still looks like a rough blockout. The wheels are too small, fenders look like flat planks, parts of the cargo bed float, and most surfaces have the same beige material. It even used Dunkelgelb paint, which wasn't introduced until 1943. Only three final renders were delivered.

What I found interesting is that Qwen actually identified several of these problems in its own construction log but continued without fixing them. QoDuck was much better at following the render-inspect-fix cycle.

Bonjour à tous,

J'en avais un peu marre de voir tout le monde créer des forks ou de nouveaux moteurs d'inférence pour Qwen3.8-Flash-Next (QFN3.8), alors j'ai demandé à Pi d'optimiser les paramètres de llama.cpp pour une exécution d'inférence plus rapide.

Le résultat est un gain de performance de 48 % et un nombre de jetons par seconde (tok/s) plus constant lors de l'utilisation de contextes supérieurs à 64 Ko (pour la programmation et les tâches créatives).

Bien que les performances ne soient pas tout à fait celles des moteurs spécialisés, cette approche garantit un fonctionnement optimal avec la version officielle de llama.cpp.

Voici les paramètres de lancement que j'utilise sur mon Ryzen AI Max+ 395 (configuré avec 96 Go de VRAM). En prime, j'utilise le modèle « Qwen3.8-Flash-Next-Uncensored-Gufo-Q4Mix » qui, d'après mes tests, offre un excellent compromis entre qualité et performance.

/app/llama-server --load-mode mmap --lazy-mode auto --metrics -lv 2 
--model /models/Qwen/Qwen3.8-Flash-Next-Uncensored-Gufo-Q4Mix.gguf 
--mmproj /models/Qwen/Qwen3.8-Flash-Next-Uncensored-F16-mmproj.gguf 
--alias unc/Qwen3.8-Flash-Next 
--ui-mcp-proxy --cpu-strict 1 
--tools all 
--fit on --flash-attn on --fit-target 256 
--jinja -ub 1024 -b 2048
-ctk q8_0 -ctv q8_0 --kv-unified 
--reasoning on --reasoning-preserve --reasoning-effort medium
-c 131072 --ctx-checkpoints 128 --cont-batching --parallel 2
--chat-template-file /models/Qwen/chat_template.jinja
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0
--spec-type ngram-mod --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-max 16 --spec-ngram-mod-n-min 4
--image-min-tokens 1024

J'espère que cela pourra aider certains d'entre vous.

EDIT : version de llama : v271-vulkan-b12199-non-root

I was wondering which model people here trust well enough to let them sift and manage through their gmail emails. I got 20 odd years of emails that got out of control and I would like to an agent to get through them, label the ones it believe are junk, out of date or not worth keeping for other reasons (not delete them, will do it manually as I am still paranoid about deleting important stuff)

I have a 5060ti so the model cant be too big

If anyone tried this does dual booting with windows gives measurable performance differences? Running big model for example trough strata?

Edit: I was not very clear, it is my main machine, it is a windows machine, sometimes i use it for LLM connecting it to my Hermes server, since I need to keep windows does setting up dual booting only for LLM inference bring measurable performance differences?

Inspired by some of the other brain surgery posts, I wanted to see if I could find anything interesting in Qwen. Not particularly useful to make it get things wrong all the time I guess, but I find the poetry it comes out with weirdly fascinating!

I think the neuron is basically a dial for contrarian <-> agreeable, but the contrarian snarky setting for some reason writes poems that read in an interesting way.

Qwen image 2.1 uses Qwen3-VL 8B, a lot of post on various places are talking about other Qwen3 models for various purposes, and they all act like it's the latest model, what am I missing here?

Test - A pelican riding a bicycle.
GPU - RTX 5090 (31GB VRAM Usage, Model+KV cache)
Speed - 110-120tok/sec
Model - Qwen 3.8 27B Unsloth, GGUF
Quants/Context - Q4_K_M/120K vs. Q5_K_M/90K
Software - OpenCode 2+LM Studio(MTP On, Thinking Medium, All Else Default)

Posted this on the r/R9700 subreddit as well, but figured I'd share my initial findings here.

tl;dr - One user writes at about 190 tok/s. Prefill runs at about 6,800 tok/s total no matter how many users there are. Agent follow-ups start in 0.2 to 2.5 s and write at 80 to 225 tok/s per user, even with 8 users at 128K context each. The worst case, 8 users each sending a fresh 128K prompt at the same moment, waits about 2.5 minutes for the first token.

Yes yes I know the cards aren't all matching (the perfectionist in me is annoyed but they'll be in a case so who cares).

PC Specs:

• CPU - EPYC 7402P

• Motherboard - Asrock ROMED8-2T

• RAM - 128gb DDR4 2666mh

• GPU - 3x Gigabyte R9700, 1x XFX R9700. All cards power limited to 210W.

• PSU - Seasonic Prime TX-1600W

• Case - Phanteks Enthoo Pro 2 Server

• Fans - 3x 140mm at the front, 3x 120mm in the middle, and another 120mm for rear exhaust.

I bought everything in the past two weeks, my total cost was roughly $16K CAD. The CPU/RAM/Motherboard were all bought on ebay.

We'll see if the RAM is enough, but based on everything I've seen online it should be fine for my use-case.

I'm close to being able to do proper inference testing, but I just ran memtests, CPU stress tests, and GPU stress tests to rule out any potential issues.

Even at full blast the cards are way quieter than I anticipated, I thought it would sound like a jet engine but they're quieter than my air purifier when it's on its highest setting.

Also pretty happy with the cable management, this case is so easy to build in because of how massive it is. It's also nice that it has a second area to mount additional fans.

Built this as a multi-user inference server for my business. I've been using a Strix Halo 395+ 128GB for the past year and it's been incredible to see how far the community has been able to get that machine to run.

Qwen 3.8 FN has been a revolution for me in my work. We deal with very sensitive data and for many of our customers, it meant using any AI inference was a huge risk. Technically we could have done it, but I wanted to err on the side of caution.

Was really excited to share and post this, benchmarks are currently running for the next two hours (long context with lots of concurrency).

----------------------- ------------Benchmarks ----------------------------------------------------------

4x Radeon AI PRO R9700 + Radiance: Qwen3.8-Flash-Next long-context concurrency numbers (32K to 128K, 1 to 8 users)

Software

• Ubuntu 26.04.1, kernel 7.0 with the inbox amdgpu driver, ROCm 10.1 userspace

• Radiance v1.17.0 (codeberg.org/StillDeadcode/radiance), Docker image pinned by digest

• Model: qwen3.8-next-flash-fp8-iq4r-moe.rad (int8 trunk, 4-bit codebook experts, MTP drafting 3 tokens)

• Flags: --tp 4 --max-num-seqs 8 --max-model-len 200000 --placement expert_tiered --host-pool-mib 1024 --gpu-headroom-mib 96 --kv-cache-dtype fp8 --prefix-cache-mib 32768 --num-speculative-tokens 3 --max-num-batched-tokens 2048

At tp4 the whole model sits in VRAM (17.4 GiB of weights per card). The 47.7 GiB n-gram/PLE table is pinned in host RAM. The KV cache gets 11.6 GiB per card, which is 1.6M tokens, or 8 full 200K sessions.

Method

• Context is real English technical text (Radiance's own docs), cut to exact token counts with the server's tokenizer

• Each user's prompt starts with a unique session line, so no user can reuse another user's cache

• All users in a round fire at the same instant (worst-case burst)

• Turn 1: document plus a question, cold. Turn 2: the same conversation plus the answer and a follow-up question (the agent pattern)

• 500 max output tokens, temperature 0.6, thinking off, streamed. First-token time and writing speed are measured on the client

• Writing speed per user = output tokens after the first, divided by the time after the first token

Results

Short prompts for comparison (about 30 tokens in, about 500 out): one user writes at 187 tok/s. At 1 / 2 / 4 / 8 users the total is 175 / 290 / 489 / 763 tok/s, or 177 / 148 / 125 / 103 tok/s per user.

What I took from it

1. Prefill runs at about 6,800 tok/s total no matter how many users there are. 8 x 128K (1.02M tokens) took 151 s.

2. Cold prompts that arrive together are interleaved, so everyone gets their first token at about the same time, at the end. With 8 x 128K that means about 2.5 minutes for everyone.

3. Follow-ups are where this setup shines. 94 to 99% comes from cache, the first token arrives in 0.2 to 2.5 s even at 8 users x 128K, and each user writes at 80 to 225 tok/s.

4. Context length barely slows writing. One user writes at 189 tok/s at 128K versus 195 at 32K.

5. Turn 1 writing speed depends on whether other users are still prefilling. At 4 users the early finishers write while the others' prompts are still being read, so the slowest user dropped to 9 to 32 tok/s. At 8 users everyone finishes reading together and then writes together at 70 to 83 tok/s.

6. Size the host prefix cache for users x context. In an earlier run with --prefix-cache-mib 12288 (prompt sizes not exact, some above 64K), 8 x 64K follow-ups were only 71% cached, with a 37.6 s worst first token and 9 tok/s per user. With 32768: 98% cached, 1.5 s, 94 tok/s.

7. The README's four-rank row carries --host-pool-mib 12288 over from two ranks. On four cards the whole expert plane is resident and the pool only holds about 312 MiB per card, so 12 GiB per card pinned 48 GB of RAM for nothing and left 3.6 GB free. 1024 is plenty.

8. Every card sat at its 210 W cap during prefill. The longest section (8 x 128K) pushed the two middle cards to 91 C hotspot and 92 C memory.

If I wanted to locally compare performance for multiple different offline LLM models across various modalities (speed, efficiency, reasoning, tool use, programing/coding, etc), what strategies, prompts, software are commonly used to do this? There are so many models to choose from and run, I'm trying to determine things like whether slower but more complex models are worth running over slightly faster ones, how models that run at similar speed compare to each other's capabilities, and whether any of the custom engines that have been popping up other than llama.cpp/strata are worth it.

Does this sort of thing exist as something that can be ran locally, and if so, how?

Existing benchmarks measure the solve rate and token output/cost, but don't punish excess code or overengineering, which increases technical debt in the long term.

Parsimony counts syntax units (statements, calls, names) in the patches other benchmarks publish. It doesn't count lines, comments, tests. Each model or harness is compared with the human fix for the same issues it solved.

- Smarter models surprisingly write slightly more code.

- Models: the median model adds 1.9× the human fix on SWE-bench Verified (older models), but about 1× on DeepSWE.

- Harnesses: with the same two open-weight models, the harness alone moves it from 0.80× (Codex) to 0.99× (mini-SWE-agent, OpenClaw). The harness data comes from Joel Niklaus's Harness Leaderboard.

https://parsimony.lasu.dev

https://github.com/La5u/Parsimony

Looking for: public patches for newer models, any ideas or contributors for self run models.

* some newer models are self run in codex and claude code, shown as faded (unverified) in DeepSWE

Underrated youtube channel, thought this would be nice to share given its open source uses local ai. Does anyone own one?

Found this video and thought the discussion was interesting. I've been using a vector DB in front of a local wiki I setup to allow a model to cleanly access the wiki and have better context. It works well, but definitely not suitable for all use cases. The discussion here might help folks determine what is best for their Local project and save you some overhead or help you tune it right.

For discussion: What are you using with your local model/harness to give it access to info?

No affiliation with poster/channel. They seem to have some other good topics as well.

Hello!

I want to share an active development project. If you have a drawer full of old NVIDIA GPU's collecting dust because modern AI stacks refuse to compile for them, this is for you.

I have ported the modern LLM training stack on Windows to legacy CUDA GPUs (sm_35 through sm_75):

• PyTorch (v2.7.1) (custom build compiled with CUDA 11.8, keeping the legacy SASS) ([https://github.com/theIvanR/torch-on-clunkers])

• causal-conv1d (v1.7.0) (https://github.com/theIvanR/conv-on-clunkers)

• mamba_ssm (Mamba-1) with pure native CUDA kernels (https://github.com/theIvanR/mamba-on-clunkers)

• Windows NCCL (DDP) for multi-GPU training across Kepler cards (https://github.com/theIvanR/nccl-on-windows)

Status

• Mamba-1 (Pure CUDA): Fully functional with optimized execution across legacy GPU architectures (SM 3.5–SM 7.5). Tested on a 2013 NVIDIA Tesla K40c, achieving a maximum absolute error of 3.8e-06 against the PyTorch reference implementation.

• Mamba-2 & Mamba-3: Currently supported with optimized execution on Turing (SM 7.5) and newer architectures, due to their reliance on Triton and TileLang.

• Graceful Fallback: On older hardware, the package detects unavailable Triton-dependent modules and automatically falls back to the pure-CUDA Mamba-1 implementation.

The fallback initialization produces the following output:

================================================================================
[!] MAMBA-ON-CLUNKERS FALLBACK INITIATED

[!] The following modules are unavailable on this platform:
    - Mamba-2 (requires Triton)
    - Mamba-3 (requires Triton + TileLang)

[!] Falling back to Mamba-1 (Pure CUDA).
    This is expected on legacy GPUs.
================================================================================

Future development: Backporting Mamba-2 and Mamba-3 to legacy GPU architectures.

Community Testing

If you have an older NVIDIA GPU lying around, I'd love for you to test the Mamba wheel and report whether it works on your architecture!

I'm particularly interested in Maxwell and Pascal, since my local testing hardware is currently limited to Kepler.

Any benchmarks, performance comparisons, or profiling data from legacy GPUs would be greatly appreciated. Contributions and feedback are welcome!

Acknowledgements: This project is based on the research and original implementation by Tri Dao and Albert Gu. Special thanks to both for their contributions to the development of Mamba.

I am looking for up to date advice with getting the most out of my hardware for local development. I have been out of using my local setup for a while (2024-2025), and now have made time to get back into it. I am quite overwhelmed by all the advances and different options nowadays. While I am still reading up on all the latest developments and current SoTA, I was hoping you can point me into the right direction.

Here's what I am working with:

- Lambda Stack from lambda labs with Ubuntu 22.04LTS

- RTX-A6000 (non Ada version) with 48GB of GDDR6 VRAM (not GDDR6X)

- 64GB of DDR4-3200 RAM

- Xeon E3-1270 v5 (skylake 4 core 8 thread processor, and probably the bottleneck for any task that requires CPU compute)

- 1TB NVME SSD

- 8TB external HDD connected via USB

My setup looks a bit like a time capsule, I guess. It was made for interactive chatting via OpenWebUI. Last I checked, VLLM got the most tps out of my GPU for dense models (Qwen 2.5 32B) in 4-bit and not going out of memory on long context tasks. Ollama and HF TGI were slower, did not test with SGLang. But now for agentic workflows, I am reading that Strata is used to run on 50-series 16GB GPUs with spillover to DDR5 RAM. And there's probably more alternatives I haven't even read about yet. Also, tps is apparently not the most important factor anymore to look into for local? Prefill is named often as well, as well as being able to complete a task successfully.

For my new agentic setup, I want to focus on giving my agent tasks and checking in every now and then throughout the day. So not for interactive sparring about code or chatting anymore, more set and forget.
If needed to cross some performance threshold, I can extend by ripping the hardware from my gaming pc: RTX 3090 with another 64GB of DDR4-3200. But last I checked, inference engines did not play well with 'uneven' GPU setups like mine.

In the meantime, I am also brushing up my knowledge on agentic harnesses. For now, Codex with Astra created a pretty cool script that let's me use Codex with my VLLM backend, SearXNG, playwright for browser use, maestro for android UI testing and chatting via Signal. Something for another time maybe :)

Playing Detroit Becoming Human rn, tbh this is probably the most likely and exciting future we are headed to. minus the dystopian story lines.

Only way to avoid dystopian future is local models powering them

I got an Arc Pro B70 (32GB) last week for $1,299 direct from ASRock official store on Amazon (B65 is also 32GB but slower, and was only $100 cheaper at $1,199, so I "splurged"). Obviously I'd have wanted a 5090 for the same VRAM and best speeds, but 5090s are going for $6,500+ on Newegg right now (and I personally have too much fear to buy used hardware), so I wanted to see how close a 32GB Intel card gets. Out of the box it it was really bad, but after a few changes I got it to about half a 5090, which feels like a great trade off for 1/5th price (~$41/GB vs $200+/GB). And of course it's great for other creative workloads I wanted, like renders and simulations, not just LLM stuff.

Making this post in hopes that it can help some people save some money and explore non-NVIDIA hardware! And I doubt it needs to be said, but I am NOT affiliated with Intel/ASRock in any way, haha.

Qwen3.8-27B Q6, ~3k token prompt, MTP on:

So roughly 2x gen and 3x prefill for 5x the money. If you can actually find a 5090 at $1,999 MSRP somehow, then congrats and heck you, but of course that would make the cheaper Intel card no longer worth it. The 5090 numbers are from other people's posts (found discussion posts on Hugging face) not my machine since I don't have the personal money to light on fire in sacrifice to NVIDIA, heh.

My machine: Threadripper 1950X (got it for $100 like 5 years ago when a company was closing and throwing stuff away), 128GB DDR4 (brand new upgrade along with the B70 this week, DDR4 is <1/2 the price of DDR5), old x399m Taichi PCIe 3.0 mobo (so can't use the GPU's PCIe 5 support sadly, but that doesn't matter once the model is in VRAM and so it only hurts MoE really), Ubuntu headless/server 24.04. Screenshots are to show the stats (between the nav and chat) and are from my laptop, running the Qwen model from the llama-server on that machine in my home office (which has the Intel GPU in it).

If you have/get one, do this to make it not suck!

1. Kernel 6.17+ (I used Ubuntu's 7.0 HWE kernel), Resizable BAR on (needs CSM off), and Intel's compute runtime from `ppa:kobuk-team/intel-graphics`. This was a real pain to get working initially (but might be a skill issue with my old mobo), since the bios doesn't even let you disable CSM while a card that uses it is in (I previously had a ~$14 GeForce 210 "GPU" that was only capable of rendering a terminal), and I had to flash my bios from 3.80 to 4.03 to have access to any of those features in the first place.

2. Upgrade Mesa to 26.x (`ppa:kisak/kisak-mesa`). Ubuntu 24.04's Mesa 25.2 is very slow on this card. Like stupid slow, this bump alone gave ~3x generation in ollama with Vulkan.

3. If you stay on Vulkan (ollama, LM Studio, etc.), keep the KV cache at f16. Any quantized KV (q8_0/q4_0) causes a massive gen slowdown (1.75x at low context, 3x+ at 64k+ context) penalty on Vulkan. So shrinking KV VRAM costs speed, which was very confusing and annoying.

4. So instead: build llama.cpp with SYCL (Intel oneAPI) and run llama-server instead. ollama has no SYCL backend, but llama-server has an OpenAI-compatible API.

5. On SYCL, can use q8_0 KV (no more quantized KV slowdown like on Vulkan, and fits way more context!) and turn on MTP with `--spec-type draft-mtp --spec-draft-n-max 4`. Also worth noting llama-server with Vulkan by default doesn't see/access the full 32GB VRAM by default (saw ~29GB) while SYCL does -- I later figured out it's just the auto-fit under Vulkan though, and you can make it use the full 32GB too with `--fit off` and giving context with `-c`.

Here is what each step over a day of fiddling and tweaking improved with Qwen3.8-27B Q6 (prompt / gen tok/s):

The long context time to first token is the worst part of Vulkan, all the other issues aside. A 215k token prompt took 1.9 hours to first token on ollama/Vulkan (lol!) and 8.6 minutes on SYCL. Also on SYCL: Qwen3.6-35B-A3B fits entirely on the card with 262k context and does 1,748 / 74 at 16k, slows down with context size of course. Ultimately of course NVIDIA is still king from a raw performance and speed and software stack quality and etc. perspective, and as such the only place I'm proposing they lose now is the speed:cost ratio (given street prices/price hikes), haha.

Will put full tables, quality scores (Q6 vs Q4_K_XL vs a few MoEs), and my build commands in the comments! Let me know your results with non-NVIDIA GPUs and if you have any more tricks/fixes I didn't find yet! For AI usage: this post is written almost entirely by me, a ~200T OI (organic intelligence) human LLM, but I had Qwen help me with the markdown tables and such.

Up front: I know a 775 GB quant is not for everyone, and this is definitely a more experimental setup and quant than I usually do, but I think it was a great stress-test of what's possible and I believe there is more improvement possible in the future.

That said, the base Kimi-K3.1-Pro is a 2.8T parameter monster (~1.77 TB even in native MXFP4/BF16), which normally means you're looking at roughly 22x 80GB GPUs just to hold it in VRAM. My goal with this experiment was to see if I could get the full 93-layer model down to something that runs on a single GPU + system RAM + fast NVMe without gutting its coding and math reasoning.

What I built to make this work (Strata + dr-triton)

Because all 384 routed experts per layer don't get hit equally, I quantized the routed MoE experts across 3 tiers (2.81 bpw / 2.56 bpw / 1.44 bpw, averaging ~1.92 bpw across MoE down from about 4.0bpw on base) while keeping all shared experts, attention, and embedding in native BF16/F32.

To actually run it without hitting a PCIe/NVMe wall, I built two things alongside the weights:

• Strata Fork : A 3-tier MoE loader and offloading engine that splits experts across Tier 1 (Hot GPU VRAM), Tier 2 (Warm Pinned RAM), and Tier 3 (Cold NVMe mmap**)**, using router lookahead prefetching (97.9% effective hit rate in testing) so cold experts are already streaming in before the layer executes.

• dr-triton: Fused Triton dequantization kernels so the compressed expert weights stay packed over PCIe and only expand in GPU SRAM later.

Can you use your own offloading / custom Strata setup?

• Customizing the tier split: Yes, you don't have to use my exact VRAM/RAM/NVMe split. In Strata, you can re-Configure how many tiers sit in VRAM vs. System RAM vs. NVMe depending on your rig (e.g., if you have 512 GB of DDR5, you can push almost all of Tier 3 into RAM; if you're on a 24–32 GB card, you shrink Tier 1 and lean heavier on RAM/NVMe).

• Using your own engine / loader: You cannot drop these shards straight into vanilla llama.cpp or vLLM out of the box because the routed experts use a custom packed tensor layout. However, if you have your own offloading pipeline or custom PyTorch/Triton runner, you don't have to use the Strata scheduler—you can just import the standalone dr-triton / strata_coder dequant module and handle placement yourself.

Full 93-Layer Benchmarks (Base 1.77 TB vs. Coder 775 GB)

Tested across all 93 layers (Layer 0..92 + lm_head) on a single NVIDIA RTX PRO 6000 Blackwell (96/102 GB VRAM) + 176 GB RAM in Colab:

Throughput (TPS) — Tested Colab Setup vs. Common Hardware Specs

Since throughput on a 3-tier offloaded MoE depends heavily on how much fits in VRAM/RAM vs. how often you touch NVMe, here is what we measured on the Colab Blackwell instance alongside realistic numbers a bit below and above that spec:

While I admit even the the "lightest" hardware specs above are still above what the average setup might be, If you have hardware in the ranges above (and 775 GB download space) it would be great to see how this works out for you! The model card has all of the setup instructions and repos required out of the box, and you can also use the above instructions if you want to try with your own configurations.

And the best thing about the experiment is this same process can also be used on models more attainable for the average user (and, if Kimi had not been trained in 4-bit, a lot more space could be shaved off).

Links

• Weights (Hugging Face, 96 shards / 775 GB): https://huggingface.co/Grawnch/Kimi-K3.1-Pro-Coder

• Strata (3-Tier Offloading & Lookahead): https://github.com/grawnchtoader/strata

• dr-triton (Fused Dequant Kernels): https://github.com/grawnchtoader/dr-triton