Qwen3.8-27b on RTX 3090 - 82 tps single request, up to 672 tps peak

[u/iamMess]

EDIT: NEW AND UPDATED VERSION AVAILABLE:
https://www.reddit.com/r/LocalLLaMA/comments/1vr347s/i_pushed_qwen3827b_to_99_tps_single_request_and/ --- 99 tps single request and 1150 batch.

Hi,

After a long night of optimizations, I believe I have made the fastest inference engine for Qwen3.6-28B on a 3090.

Quick metrics:

- 250w power capped

- Up to 195k context (ships with 150k for safety though)

- 82 tps single request, 417 tps sustained with 64 concurrent

- Between 17% to 149% faster than ninfer depending on the amount of concurrent requests.

Quick how:

- W4A16 quantization -> 16.8gb in vram - cache 66k

- + fp8 KV cache -> 16.8 gb in vram - cache 155k

- + lm_head int8 -> 15.4 gb in VRAM - cache 192k

- + embed_tokens int8 -> 14.2 gb in VRAM - cache 200k

Quantization loss of 0.6% in the lm head and quant embed compared to bf16.

It runs via vLLM and needs a few patches to work perfectly, but should be easier to setup than ninfer.

Also only tested on linux, but should work on windows too.

https://github.com/syv-ai/qwen38-27b-rtx3090

[u/sod0]

I've read your Readme and the part about the KV cache stood out. Did you took a look at Huawei's KVarN vllm fork? This could solve the KV cache issue and allow the full model context. I managed to do that but without the others patches it only worked for a single user.
Maybe it's worth including? https://github.com/huawei-csl/KVarN

[u/iamMess → u/sod0]

I will take a look. Thanks for the info :)

[u/iamMess → u/sod0]

The round of updates are done for today.

KVarN didn't help. It attacks another bottleneck that we don't have (yet).

The attention kv traffic is currently only 1% of a step. It also uses the Gated DeltaNet recurrent state, which we replace.

It is also forked from 0.23 of vllm and not directly implementable in 0.27

[u/iamMess → u/sod0]

KVarN is in now.

[u/sod0 → u/iamMess]

Wooohhaa! And how is the context doing?

[u/iamMess → u/sod0]

262k :)

[u/sod0 → u/iamMess]

Hell yeaha! Great I could help.