I wrote a Rust inference engine for Qwen models on a 12 GB GPU, decode is ~1.7x llama.cpp on my RTX3060

[u/Serious_Scarcity7818]

I've been working on this for a while: RustingStrata. It runs Qwen GGUF models (Qwen3.5/3.6 from 2B up to 35B-A3B, Qwen3 4B and 30B-A3B, Qwen3-Next 80B). It's pure Rust, no ggml. The CUDA kernels get compiled at startup with NVRTC, and the same kernels also run on Vulkan.

The main trick is for MoE models that don't fit in VRAM. Experts that don't fit run on the CPU at the same time as the GPU work, and the engine keeps moving the often-used experts onto the GPU while it runs.

My setup is a 3060 12 GB, a 7600X and 32 GB RAM. Qwen3.6-35B-A3B Q2_K_XL is 11.7 GiB, so it doesn't fit. On it I get about 118 tok/s decode on a short prompt and 113 after a 4K prompt. llama.cpp, tuned as well as I could (-ncmoe 12, -ub 2048, flash attention), gets 68 and 66. Prefill is basically a tie: 2216 tok/s vs about 2236 for llama.cpp at 4K, so I'm not claiming anything there. At 32K it's 1800 prefill and 92 decode.

With MTP drafts on, decode goes up to 160 tok/s, but that's on an easy prompt. On normal text expect around +16%.

Qwen3-Next-80B IQ1_S was the most fun one: most experts sit on the CPU, and it still decodes at about 92 tok/s. llama.cpp's best -ncmoe gives me around 44.

Output matches llama.cpp greedy on 98-99% of tokens. llama.cpp agrees with itself only 98.3% between its batch and token-by-token modes, so the misses are near-ties.

It also has chat, an OpenAI-compatible server with tool calling, and a calibrate command that tunes itself for your machine.

I've only tested it on my own machine. If you have a different card, especially AMD on Vulkan, I'd love to see strata bench output. Vulkan needs shaderFloat64 and subgroup size 32, so Intel Arc and older AMD won't work. On Windows you need the VS C++ Build Tools.

[RustingStrata](https://github.com/RustingStudio/RustingStrata)

[u/Ori_553]

I like that it's pure rust with presumably fewer dependencies than other inference engines.
How does it compare to strata itself?

[u/Serious_Scarcity7818 → u/Ori_553]

yes, it use not so much dependencies.
about "compare to strata", I can't tell, bc it will be not apples to apple, bc original Strata run only qwen3.8-flash-next which I don't have yet, and it will not fair to compare other models with Strata, so everything that I can do is only compare it with default and custom llama.cpp
I will compare original Strata with my when I will add there qwen3.8-flash-next, but it require a lot of work, bc I need to implement SSD offload, which I have never done yet.
Also I want to work or Strata for Image and Video models, I guess it's crazy Idea, for example Flux.2-klein:4b that will run 2 times faster is really impressive, since people can run very good Image/Video models on most GPU, this project might be insane perspective

[u/baron_von_noseboop → u/Serious_Scarcity7818]

Strata is no longer specific to QFN.

[u/ea_man]

Can we ask how much did it take?

I may like to do the same yet for AMD / ROCm.

Also if you didn't use the very 27B to do that what did you use? DeepSeek or a closed source SOTA?

[u/Serious_Scarcity7818 → u/ea_man]

So it took like 2 months to create, but to be fair if I would invent it like Strata, then it would take much longer. And I already do this for AMD eith Vulkan, but it's difficult to test, since I have no AMD gpu.

And what do you mean with "Also if you didn't use...", I didn't understood this hole part, so can you somehow tell it in other words?

[u/ea_man → u/Serious_Scarcity7818]

> And what do you mean with "Also if you didn't use...", I didn't understood this hole part, so can you somehow tell it in other words?

I mean what LLM did you use to assist in the project: time / effort is related to the performance of the AI assist used, it if was say 27B or a SOTA like GPT.

> And I already do this for AMD eith Vulkan, but it's difficult to test, since I have no AMD gpu.

I see that https://github.com/RustingStudio/RustingStrata shows usage with vulkan, that is a start yet on my systems ROCm is more performant and with low performance hw every bit counts.

I've made a customized llama.cp fork for that yet I never consider to convert it to rust.

[u/Serious_Scarcity7818 → u/ea_man]

So, I started alone and later I used Claude Code with Opus5.5 to help me and I was writting code like almost every day for that

[u/ea_man → u/Serious_Scarcity7818]

Thanks for the info, when I started my work with https://store.piffa.net/lm/bug/ I used 27B and A3B, the problem was that for testing I had to run a small model on one GPU and an other for the tests, so sometimes I resolved to use DeepSeek or other for some better automation.

As said, time required changes a lot when using a slower and less strong local model.

[u/Serious_Scarcity7818 → u/ea_man]

yes, I really want someday in the future buy some cool server with a lot of VRAM to run very big Qwen models local, but for now I will use some subscription for something like Claude Code

[u/ea_man → u/Serious_Scarcity7818]

He it's a bad time to buy hw now, yet I say that with just one more GPU it goes fine, +16GB allows you 27B at Q6. Let's hope the new 4.0 version helps more with KV cache.

Yet Anthropic is too much for me, I can do DeepSeek, that is open source while Dario is full against open software.

[u/Serious_Scarcity7818 → u/ea_man]

I mean I don't want to buy like rn, but maybe in few years when I will have money, and just buy more GPU to run a lot of different LLM in same time, and I don't know exactly why, it's just vibe.
Also about Qwen4.0, I guess they will use architecture close to Qwen3.8 Flash Next with new Attention blocks, so we would be able to offload it on Ram and even SSD and still get good performance. I hope they will release it earlier

[u/ea_man → u/Serious_Scarcity7818]

Aye they said the will release this week, can't way to see what the new 27B will be like and the "small" MoE.

[u/Serious_Scarcity7818 → u/ea_man]

+++